Mailman 3 April 2013 - Python-ideas

Re: [Python-ideas] [Python-Dev] relative import circular problem
by Guido van Rossum April 5, 2013

April 5, 2013

Redirecting to python-ideas. On Thu, Apr 4, 2013 at 9:26 AM, Richard Oudkerk <shibturn(a)gmail.com> wrote: > On 04/04/2013 4:17pm, Guido van Rossum wrote: >> >> I don't really see what we could change to avoid breaking code in any >> particular case -- the burden is up to the library to do it right. I >> don't see a reason to forbid any of this either. > > > How about having a form of relative import which only works for submodules. > For instance, instead of > > from . import moduleX > > write > > import .moduleX > > which is currently a SyntaxError. I think this could be implemented as > > moduleX = importlib.import_module('.moduleX', __package__) We considered that when relative import was designed and rejected it, because it violates the expectation that after "import <some string>" you can use exactly "<some string>" in your code. -- --Guido van Rossum (python.org/~guido)

2 1

str-type enumerations
by Ethan Furman April 4, 2013

April 4, 2013

I'm not trying to beat this dead horse (really!) but back when the enumeration discussions were going fast and furious one of the things asked for was enumerations of type str, to which others replied that strings named themselves. Well, I now have a good case where I don't want to use the string that names itself: 'Mn$(1,6)' I would match rather write item_code! :) -- ~Ethan~

3 9

zip_strict() or similar in itertools ?
by Wolfgang Maier April 4, 2013

April 4, 2013

Dear all, the itertools documentation has the grouper() recipe, which returns consecutive tuples of a specified length n from an iterable. To do this, it uses zip_longest(). While this is an elegant and fast solution, my problem is that I sometimes don't want my tuples to be filled with a fillvalue (which happens if len(iterable) % n != 0), but I would prefer an error instead. This is important, for example, when iterating over the contents of a file and you want to make sure that it's not truncated. I was wondering whether itertools, in addition to the built-in zip() and zip_longest(), shouldn't provide something like zip_strict(), which would raise an Error, if its arguments aren't of equal length. zip_strict() could then be used in an alternative grouper() recipe. By the way, right now, I am using the following workaround for this problem: def iblock(iterable, bsize, strict=False): """Return consecutive lists of bsize items from an iterable. If strict is True, raises a ValueError if the size of the last block in iterable is smaller than bsize. If strict is False, it returns the truncated list instead.""" it=iter(iterable) i=[it]*(bsize-1) while True: try: result=[next(it)] except StopIteration: # iterator exhausted, end the generator break for e in i: try: result.append(next(e)) except StopIteration: # iterator exhausted after returning at least one item, # but before returning bsize items if strict: raise ValueError("only %d value(s) left in iterator, expected %d" % (len(result),bsize)) else: pass yield result , which works well, but is about 3-4 times slower than the grouper() recipe. If you have alternative, faster solutions that I wasn't thinking of, I'd be very interested to here about them. Best, Wolfgang

3 9

Re: [Python-ideas] [Python-Dev] A bit about the GIL
by Trent Nelson April 4, 2013

April 4, 2013

On Tue, Apr 02, 2013 at 05:40:07PM -0700, Alfredo Solano Martínez wrote: > > (There are two aspects to the work; the parallel stuff, which is the > > changes to the interpreter to allow multiple threads to run CPython > > internals concurrently, and the async stuff, which will be heavily > > tied to the best IO multiplex option on the underlying platform > > (IOCP on AIX, event ports on Solaris, kqueue on *BSD, epoll on > > Linux, poll on everything else). The parallel stuff is pretty > > platform agnostic, which is nice. (Aside from the thread/register > > trick; but it appears as though most contemporary ISAs have some > > way of doing the same thing.)) > > That's a lot of things to do. Do you have a work breakdown structure or > are you still putting the pieces together? Work breakdown structure? That's far too organized ;-) I have an end goal in mind and I'm just slowly hacking my way towards it (at least for the Windows work). > > The "no refcounting and nuke everything when done" aspect has > > worked surprisingly well. Shared-nothing code executing in a > > parallel thread absolutely flies. Mallocs are basically free, > > frees are no-ops, no reference counting and no garbage > > collection; everything gets released in a single call when we're > > done. > > Glad to hear it, it's hard to make things simple. Actually, I have to > say the GPU analogy is very good, with all but the main core acting as > vector processors -and thus providing a sort of programmable pipeline > for it- while the main core becomes the CPU. I would go definitely > for that in future slides. The GPU analogy seemed like a good idea when I was writing the PEP, but the implementation has taken a slightly different path. There is far less emphasis on the notion of vectorized/SIMD-style work; in fact, I haven't implemented any of the 'parallel' type functions yet (like a parallel map/reduce, or equivalents to the parallel stuff exposed by multiprocessing). That'll be all stuff to tackle down the track. > In the case of the GPUs the copying of data from memory to card is > usually a bottleneck, is there a big hit in performance here too? Well, as the current implementation doesn't really have anything that reflects the GPU vector analogy in that draft PEP, no, not really ;-) (I should probably clarify again that the PEP I cited was hacked out in a weekend before I started a lick of coding. The requirements section is definitely useful, as it elicits the constraints I used to drive my design decisions, but all of the sections that allude to implementation details (like binding a thread to each core via thread affinity, not having access to globals, introducing new op- codes to achieve the parallel functionality) don't necessarily map to how I've implemented things now. Once I've finished the work on Windows I'll do an updated PEP.) Trent.

2 1

ProtoPEP: A Standardized Marker for Mirroring the TTY to Other Devices
by Eric Snow April 3, 2013

April 3, 2013

I received this proposal from an interested party and am passing it along for consideration. I am concerned about a conflict of interest as the author has ties to the paper industry, and thus could stand to benefit financially from the proposal, but am willing to give them the benefit of the doubt. -eric #################################### PEP: 4XX Title: A Standardized Marker for Mirroring the TTY to Other Devices Version: $Revision$ Last-Modified: $Date$ Author: Michael G. Scott <michael at dundermifflin.com> BDFL-Delegate: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/plain Created: 01-Apr-2013 Post-History: Abstract The history of computing offers a progression of text I/O devices, most notably the Teletype machine [1]. This device had such an impact that we still identify our terminals by "TTY". Presently the terminal text I/O in Python is facilitated through sys.stdin/stdout/stderr. This is a proposal to facilitate mirroring the data passing across these three objects to another device on the system such as a printer. The API to the do so would be through a new builtins special name: __mifflin__. Rationale Dumping the TTY to a printer is a natural desire for anyone who knows anything about computer history and an unrecognized longing for everyone else. Though perhaps no one remembers it this way, there was a strong outcry to the introduction of monitors and keyboards as a replacement for the teletype machine. Everyone loved the teletype. The lamentable fact that monitors and keyboards won out is due to the efforts of the monitor-and-keyboard lobby and particularly to the pathological (but successfully concealed) fear of teletypes of one Thomas Watson, Jr. [2]. Why did people love the teletype? Hard copies are much more endearing. Consider how lame e-books are. A monitor will never give you that fresh ink smell, that textured feel of paper in your hands. You can't make an airplane out of a kindle or wad up your monitor into a ball and throw it into a trash can. People try to accomplish the teletype's satisfying audible response to each keystroke with fancy "retro" keyboards, but they will never be satisfied until they hear the original. Furthermore, look to the airline industry, a paragon of stability and consistency. In critical situations they mirror their TTYs to printers. This is because the hard copies are perfect for chronological review, documentation, and reliable backup. If it's good enough for them, it should be good enough for us. Usage Much as you would with __import__, you will bind builtins.__mifflin__ to a Printer object (see below). By default __mifflin__ will be set to sys.printer (see below). io.Printer and sys.printer A new type will be added to the io module called Printer. Printer may be initialized with a file-like object that exposes an underlying printer device (by default the system printer). Printer will have a write() method that takes a string and writes it out to the underlying system printer. A new attribute will be added to the sys module named "printer". sys.printer will be a Printer object, with a default of a wrapper around the system printer. References [1] http://en.wikipedia.org/wiki/Tty_(Unix) [2] https://en.wikipedia.org/wiki/Thomas_Watson,_Jr. Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8

3 2

Re: [Python-ideas] [Python-Dev] A bit about the GIL
by Trent Nelson April 2, 2013

April 2, 2013

On Mon, Apr 01, 2013 at 04:29:59PM -0700, Alfredo Solano Martínez wrote: > > I've been making pretty good progress with my pyparallel work. See > > the initial slides here: > > > > http://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-altern… > > Really interesting stuff, thanks for the link. Is there a video of the > presentation available? Just the slides for now, unfortunately. > > I've since set up a separate mailing list for it here: > > > > http://lists.snakebite.net/mailman/listinfo/pyparallel/ > > > > And switched to bitbucket.org for the primary repo (I still commit > > to hg.python.org/sandbox/trent too, though): > > > > https://bitbucket.org/tpn/pyparallel > > Will take a look at all that. Any ETA for the Linux version? Nothing formal, no. The plan is to work out all the kinks on Windows, then step back, figure out the best way to abstract the API, then attack the POSIX implementation. (There are two aspects to the work; the parallel stuff, which is the changes to the interpreter to allow multiple threads to run CPython internals concurrently, and the async stuff, which will be heavily tied to the best IO multiplex option on the underlying platform (IOCP on AIX, event ports on Solaris, kqueue on *BSD, epoll on Linux, poll on everything else). The parallel stuff is pretty platform agnostic, which is nice. (Aside from the thread/register trick; but it appears as though most contemporary ISAs have some way of doing the same thing.)) > > You might find this interesting, too: > > > > http://hg.python.org/sandbox/trent.peps/file/6de5ed566af9/pep-async.txt > > There's a lot of nice ideas there. It reminded me of the typical MPI > workflow, with the main thread as the master process (the GIL acting > as the barrier) doing the scatter and gather to the other processes. > I really liked the part about not doing any reference counting and > just nuking everything after its done (it's the only way to be sure) You know, I re-read that PEP last night for the first time since I wrote it. I found it quite amusing -- some things are completely wacky, but quite a lot of it is pretty close to how everything is now. I had zero experience with CPython nitty-gritty internals when I wrote that, which is pretty evident from some of the things I'm suggesting. The "no refcounting and nuke everything when done" aspect has worked surprisingly well. Shared-nothing code executing in a parallel thread absolutely flies. Mallocs are basically free, frees are no-ops, no reference counting and no garbage collection; everything gets released in a single call when we're done. Factor in the heap snapshot/reset/rollback stuff in the IO loop and it's extremely cache friendly too. Definitely very pleased with how all of that stuff is shaping up. > Alfredo Regards, Trent.

1 0

Re: [Python-ideas] [Python-Dev] A bit about the GIL
by Trent Nelson April 1, 2013

April 1, 2013

On Sun, Mar 31, 2013 at 04:14:11PM -0700, Alfredo Solano Mart?nez wrote: > Hi, > > I know this may be tiresome by now, so feel free to ignore, but I'd like to > share with the list an idea about the GIL, more specifically the reference > counting mechanism. I've been making pretty good progress with my pyparallel work. See the initial slides here: http://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-altern… And follow-up thread starting here: http://mail.python.org/pipermail/python-dev/2013-March/124690.html I've since set up a separate mailing list for it here: http://lists.snakebite.net/mailman/listinfo/pyparallel/ And switched to bitbucket.org for the primary repo (I still commit to hg.python.org/sandbox/trent too, though): https://bitbucket.org/tpn/pyparallel TL;DR version: I've come up with a way to exploit multiple cores without impeding the performance of "single-threaded execution", and also coupled it with a host of async facilities that allow you to write Twisted/Tulip style protocols but have callbacks automatically execute across all cores. (In the case of client/server facilities, one neat feature is that it'll automatically switch between sync and async socket methods based on concurrent clients and available cores; there is a non-negligible overhead to doing async IO versus blocking IO -- if you have 64 cores and only 32 clients, there's no reason not to attempt sync send and recvs; this will maximize throughput. As soon as the client count exceeds available cores, it'll automatically do async sends/recvs for everything, improving concurrency (at the expense of throughput). The chargen example in the slides is a perfect example of this in action.) > Simply put, make the reference counter a sharded one. That is, separate it > into several subcounters, in this case one for each thread. > > The logic would then be something like this: > - when increasing the refcount, a thread writes only to its own subcounter, > creating one first if necessary. > - similarly, when decreasing the refcount, there is no need to access other > subcounters until that subcounter reaches zero. > - when a subcounter gets to zero, delete it, and read the other subcounters > to check if it was the last one. > - delete the object only if there are no more subcounters. > > Contention could then be reduced to a minimum, since a thread only needs to > read other subcounters when its own reaches zero or wants the total value. > Depending on the implementation it might help with false sharing too, as > subcounters may or may not be in the same cache-line. > > Unfortunately, in a crude test of mine there is already a severe performance > degradation, and that is without rwlocks. I've used a basic linked list, > and changed the INCREF/DECREF macros to functions to accommodate the extra > logic so it may not be the best approach (too many dereferences). > > Does this makes sense to anyone? My personal (and extremely biased now that I've gained some momentum with pyparallel) opinion is that trying to solve the free threading problem is wrong. That is, allowing existing Python code written to use threading.Threads() to execute concurrently across all cores. The only way that could ever be achieved is with the introduction of fine grained locking or STM-type facilities; both of which seriously impede single-threaded performance. You might find this interesting, too: http://hg.python.org/sandbox/trent.peps/file/6de5ed566af9/pep-async.txt I wrote that the weekend before I started actually coding on pyparallel. I think the only person that's seen/read it is Guido (and maybe Antoine). It goes into detail regarding the rationale for the design decisions I took (especially re: avoiding fine grained locks/STM etc). Trent.

1 0

Re: [Python-ideas] [Python-Dev] A bit about the GIL
by Charles-François Natali April 1, 2013

April 1, 2013

> I know this may be tiresome by now, so feel free to ignore, but I'd like to > share with the list an idea about the GIL, more specifically the reference > counting mechanism. > > Simply put, make the reference counter a sharded one. That is, separate it > into several subcounters, in this case one for each thread. Yeah, that's known as sloppy counters, see e.g. http://pdos.csail.mit.edu/papers/linux:osdi10.pdf, section 4.3. Actually you don't need a per-thread counter, only per-cpu (see sched_getcpu()/getcpu()), although I'm not sure it'd be as fast as using thread-register like Trent does. > Unfortunately, in a crude test of mine there is already a severe performance > degradation, and that is without rwlocks. I've used a basic linked list, > and changed the INCREF/DECREF macros to functions to accommodate the extra > logic so it may not be the best approach (too many dereferences). I think that's a dead-end. Extra indirection and cache miss will kill performance. Also, such counters are mostly useful for data structures which are only seldom deallocated (because reconciling the counters is expensive), and with garbage collection, you tend to have many short-lived objects (generational hypothesis). Finally, that'll increase the size of objects consequently (since you need a counter per core/thread), which is bad for cache. > Does this makes sense to anyone? You can try ;-) cf P.S.: Starting a thread on the GIL on April fools' day is a great idea!

1 1