Mailman 3 May 2012 - Python-ideas

PEP for Python folder structure
by Armin Wieser May 30, 2012

May 30, 2012

Hi, I would like to write a PEP about folder structure in python projects. You will think that there is no need for that, because everything is documented (package, module, setuptools). But it should contain something like [0]. If you aren't into those concepts, never have pushed some package to pypi, and you only have written some scripts, it's hard to find out how to structure your folders. Therefore i think a PEP would be a great way to show how you can do it. What do you think about it? [0] http://jcalderone.livejournal.com/39794.html

5 4

a simple namespace type
by Eric Snow May 30, 2012

May 30, 2012

Below I've included a pure Python implementation of a type that I wish was a builtin. I know others have considered similar classes in the past without any resulting change to Python, but I'd like to consider it afresh[1][2]. class SimpleNamespace: """A simple attribute-based namespace.""" def __init__(self, **kwargs): self.__dict__.update(kwargs) # or self.__dict__ = kwargs def __repr__(self): keys = sorted(k for k in self.__dict__ if not k.startswith('_')) content = ("{}={!r}".format(k, self.__dict__[k]) for k, v in keys) return "{}({})".format(type(self).__name__, ", ".join(content)) This is the sort of class that people implement all the time. There's even a similar one in the argparse module, which inspired the second class below[3]. If the builtin object type were dict-based rather than slot based then this sort of namespace type would be mostly superfluous. However, I also understand how that would add an unnecessary resource burden on _all_ objects. So why not a new type? Nick Coghlan had this objection recently to a similar proposal[4]: Please, no. No new just-like-a-namedtuple-except-you-can't-iterate-over-it type, and definitely not one exposed in the collections module. We've been over this before: collections.namedtuple *is* the standard library's answer for structured records. TOOWTDI, and the way we have already chosen includes iterability as one of its expected properties. As you can see he's referring to "structured records", but I expect that his objections could be extended somewhat to this proposal. I see where he's coming from and agree relative to structured records. However, I also think that a simple namespace type would be a benefit to different use cases, namely where you want a simple dynamic namespace. Making a simple namespace class is trivial and likely just about everyone has written one: "class Namespace: pass" or even "type('Namespace', (), {})". Obviously the type in this proposal has more meat, but that's certainly not necessary. So why a new type? The main reason is that as a builtin type the simple namespace type could be used in builtin modules[5][6][7]. Thoughts? -eric [1] http://mail.python.org/pipermail/python-dev/2012-May/119387.html [2] http://mail.python.org/pipermail/python-dev/2012-May/119393.html [3] http://hg.python.org/cpython/file/dff6c506c2f1/Lib/argparse.py#l1177 [4] http://mail.python.org/pipermail/python-dev/2012-May/119412.html [5] http://mail.python.org/pipermail/python-dev/2012-May/119395.html [6] http://mail.python.org/pipermail/python-dev/2012-May/119399.html [7] http://mail.python.org/pipermail/python-dev/2012-May/119402.html -------------------------- class Namespace(SimpleNamespace): def __dir__(self): return sorted(k for k in self.__dict__ if not k.startswith('_')) def __eq__(self, other): return self.__dict__ == other.__dict__ def __ne__(self, other): return self.__dict__ != other.__dict__ def __contains__(self, name): return name in self.__dict__

10 18

Reimplementing collections.deque as a dynamic array
by Julian Berman May 29, 2012

May 29, 2012

I've occasionally had a need for a container with constant-time append to both ends without sacrificing constant-time indexing in the middle. collections.deque will in these cases narrowly miss the target due to linear indexing (with the current use case being for two deques storing the lines of text surrounding the cursor in a text editor while still being randomly indexed occasionally). Wikipedia lists at least two common deque implementations: http://en.wikipedia.org/wiki/Double-ended_queue#Implementations where switching to a dynamic array would seemingly satisfy my requirements. I know from a bit of experience (and a quick SO perusal) that "How do I index a deque" does occasionally come up. Any thoughts on the value of such a change? JB

3 2

from foo import bar.baz
by Devin Jeanpierre May 29, 2012

May 29, 2012

Has it irritated anyone else that this syntax is invalid? I've wanted it a couple of times, to be equivalent to: import foo.bar.baz from foo import bar del foo # but only if we didn't import foo already before" The idea being that one wants access to foo.bar.baz under the name bar.baz , for readability purposes or what have you. I played around with adding this, but I seem to have really bad luck with extending CPython... -- Devin

3 2

pmap, preduce, pmapreduce?
by Mike Meyer May 26, 2012

May 26, 2012

Another crazy idea that may not be possible, based on my finally getting around to watching Guy Steele's talks about what he's up to these days (http://vimeo.com/6624203). Given a function that takes a list (or a container class which len doesn't consume) and a function, and then applies that function to the list in some way: either element wise, or in pairs of elements/results, but does it in parallel. It will hold the GIL, but run the function calls in distinct threads, meaning two applications of the function could interfere with each other. Is it possible to place limitations on the function such that this kind of controlled concurrent operation is safe? <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

4 4

Add a generic async IO poller/reactor to select module
by Giampaolo Rodolà May 26, 2012

May 26, 2012

Including an established async IO framework such as Twisted, gevent or Tornado in the Python stdlib has always been a controversial subject. PEP-3153 (http://www.python.org/dev/peps/pep-3153/) tried to face this problem in the most agnostic way as possible, and it's a good starting point IMO. Nevertheless, it's still vague about what the actual API should look like and AFAIK it remained stagnant so far. There's one thing in the whole async stack which is basically the same for all implementations though: the poller/reactor. Could it make sense to add something similar to select module? Differently from PEP-3153, providing such a layer on top of select(), poll() & co. is easier and could possibly be an incentive to avoid such code duplication. I'm coming up with this because I recently did something similar in pyftpdlib as an hack on top of asyncore to add support for epoll() and kqueue(), using the excellent Tornado's io loop as source of inspiration: http://code.google.com/p/pyftpdlib/issues/detail?id=203 http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop… The way I imagine it: >>> import select >>> dir(select) [..., 'EpollPoller', 'PollPoller', 'SelectPoller', 'KqueuePoller'] >>> poller = select.EpollPoller() >>> poller.register(fd, handler, poller.READ | poller.WRITE) >>> poller.socket_map {2 : <Handler instance at ...>} >>> poller.modify(fd, poller.READ) >>> poller.poll() # will call handler.handle_read_event() if/when it's the case ^C KeyboardInterrupt >>> poller.remove(fd) >>> poller.close() The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event Users willing to support multiple event loops such as wx, gtk etc can do: >>> while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll() Basically, this would be the whole API. Thoughts? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

8 20

Minimal built-ins (+ tiny doc suggestion)
by Mark Summerfield May 25, 2012

May 25, 2012

Hi, Built-ins: In an effort to keep the core language as small as possible (to keep it "brain sized":-) would it be reasonable to deprecate filter() and map() and to move them to the standard library as happened with reduce()? After all, don't people mostly use list comprehensions and generator expressions for these nowadays? Docs: The Python Module Index http://docs.python.org/dev/py-modindex.html Shows _ | a | b | ... This is prettier than _ | A | B | ... but also harder to click because the letters are smaller; so I would prefer the use of capitals. -- Mark Summerfield, Qtrac Ltd, www.qtrac.eu C++, Python, Qt, PyQt - training and consultancy "Programming in Go" - ISBN 0321774639 http://www.qtrac.eu/gobook.html

8 8

Composability and concurrent.futures
by Adrian Sampson May 24, 2012

May 24, 2012

The concurrent.futures module in the Python standard library has problems with composability. If I start a ThreadPoolExecutor to run some library functions that internally use ThreadPoolExecutor, I will end up with many more worker threads on my system than I expect. For example, each parallel execution wants to take full advantage of an 8-core machine, I could end up with as many as 8*8=64 competing worker threads, which could significantly hurt performance. This is because each instance of ThreadPoolExecutor (or ProcessPoolExecutor) maintains its own independent worker pool. Especially in situations where the goal is to exploit multiple CPUs, it's essential for any thread pool implementation to globally manage contention between multiple concurrent job schedulers. I'm not sure about the best way to address this problem, but here's one proposal: Add additional executors to the futures library. ComposableThreadPoolExecutor and ComposableProcessPoolExecutor would each use a *shared* thread-pool model. When created, these composable executors will check to see if they are being created within a future worker thread/process initiated by another composable executor. If so, the "child" executor will forward all submitted jobs to the executor in the parent thread/process. Otherwise, it will behave normally, starting up its own worker pool. Has anyone else dealt with composition problems in parallel programs? What do you think of this solution -- is there a better way to tackle this deficiency? Adrian

3 4

Extended "break" "continue" in "for ... in" block.
by 海韵 May 24, 2012

May 24, 2012

Hi all... i'd like to propose a syntax extend of "break" and "continue" to let them work together with "yield". Syntax: continue_stmt: "continue" [ test ] break_stmt: "break" [ test ] it is only valid in "for ... in" block. when we writing "for <fields> in <container>: <suite>", we can say there is a generator ("__g = iter(<container>)") providing values ("<fields> = next(__g)") and the values are processing by <suite>. <suite> implenetments computing. (we focus) inside geneator("__iter__" of <container>) implenetments iteration. (sealed logic) ---- as current ---- "continue" is "next(__g)" (which equals to "__g.send(None)"), "break" leave the block and __g is garbage collected(which implies a __g.close()). "return" "raise" inside <suite> leave the block and __g is garbage collected. let's make thing reverse. consider we are write __g' code. generator function implenetments computing. (we focus) outside code(<suite> of "for ... in") implenetments continuation. (sealed logic) ---- as proposal ---- "continue <test>" is equiv to "__g.send(<test>)". "continue" is alias of "continue None". "break <test>" is equiv to "__g.throw(<test>)". "break" is alias of "break GeneratorExit". "return" "raise" inside <suite> impies a "break" to __g, with communication between "yield" and "continue", "break", this plays just as Ruby's block except return value of __g lost (may we use an "as" after "for ... in" to fetch return value?). -- = =!

4 3

[...].join(sep)
by anatoly techtonik May 23, 2012

May 23, 2012

I am certain this was proposed many times, but still - why it is rejected? "real man don't use spaces".split().join('+').upper() instead of '+'.join("real man don't use spaces".split()).upper() The class purity (not being dependent from objects of other class) is not an argument here: string.join() produces list, why list.join() couldn't produce strings? The impedance mismatch can be, but it is a pain already and string.join() doesn't help: that means you still get exception when trying to join lists with no strings inside Can practicality still beat purity in this case?

6 6