Mailman 3 January 2014 - Python-ideas

Predicate Sets
by Daniel da Silva 21 Jan '14

21 Jan '14

Below is a description of a very simple but immensely useful class called a "predicate set". In combination with the set and list comprehensions they would allow another natural layer of reasoning with mathematical set logic in Python. In my opinion, a concept like this would be best located in the functools module. *Overview:* Sets in mathematics can be defined by a list of elements without repetitions, and alternatively by a predicate (function) that determines inclusion. A predicate set would be a set-like class that is instantiated with a predicate function that is called to determine ``a in the_predicate_set''. >> myset = predicateset(lambda s: s.startswith('a')) >> 'xyz' in myset False >> 'abc' in myset True >> len(myself) Traceback (most recent call last): [...] TypeError *Example Uses:* # Dynamic excludes in searching foo_files = search_files('foo', exclude=set(['a.out', 'Makefile'])) bar_files = search_files('bar', exclude=predicateset(lambda fname: not fname.endswith('~'))) # exclude *~ # Use in place of a set with an ORM validusernames = predicateset(lambda s: re.match(s, '[a-zA-Z0-9]+')) class Users(db.Model): username = db.StringProperty(choices=validusernames) password = db.StringProperty()

12 17

Tail recursion elimination
by musicdenotation＠gmail.com 21 Jan '14

21 Jan '14

On April 22, 2009, Guido van Rossum wrote: > First, as one commenter remarked, TRE is incompatible with nice stack traces: when a tail recursion is eliminated, there's no stack frame left to use to print a traceback when something goes wrong later. This will confuse users who inadvertently wrote something recursive (the recursion isn't obvious in the stack trace printed), and makes debugging hard. Providing an option to disable TRE seems wrong to me: Python's default is and should always be to be maximally helpful for debugging. What are "nice" stack traces? If you mean stack traces that record every function call then it is not nice and helpful at all given their length. Do loops have nice stack traces as you mean it? No. When something goes wrong in a loop, you don't get to see every iteration in the stack trace. You debug loops by examining the values of variables in the iteration it goes wrong. Likewise you debug a tail recursive function by examining the arguments that went into the function call that blows up. And if you don't want to turn on TRE by default, you can turn it off and offer an option to enable. > Second, the idea that TRE is merely an optimization, which each Python implementation can choose to implement or not, is wrong. Once tail recursion elimination exists, developers will start writing code that depends on it, and their code won't run on implementations that don't provide it: a typical Python implementation allows 1000 recursions, which is plenty for non-recursively written code and for code that recurses to traverse, for example, a typical parse tree, but not enough for a recursively written loop over a large list. Yes, it is more of a language feature than an implementation feature. But once CPython implements it, I think other implementations will follow suit or Python developers will not write code that uses TRE just like JavaScript developers don't use Mozilla-specific extensions like let keyword. > Third, I don't believe in recursion as the basis of all programming. This is a fundamental belief of certain computer scientists, especially those who love Scheme and like to teach programming by starting with a "cons" cell and recursion. But to me, seeing recursion as the basis of everything else is just a nice theoretical approach to fundamental mathematics (turtles all the way down), not a day-to-day tool. This isn't a valid argument. That something isn't fundamental is almost never an argument to leave it out. (Except for Scheme.) > Of course, none of this does anything to address my first three arguments. Is it really such a big deal to rewrite your function to use a loop? (After all TRE only addresses recursion that can easily be replaced by a loop. :-) It isn't a big deal, yes. But why restrict programmers? Python is a multi-paradigm language anyway: you can write imperative or functional code with(out) object orientation. "There should be one – and only one – obvious way to do it"? You are misinterpreting it. It is about having only one obvious way to do things, but you have removed another obvious way to do certain things because certain problems are better expressed using recursion rather than looping (traversing a tree, for example). I think the most obvious way to do something is how it is defined. If there was really one way to do anything, there should be no for-loops, list comprehensions, or even object orientation at all in Python. Remember that recursion is either something lower- or higher-level than looping. And if you don't like abstractions over primitive concepts, you should be coding in machine code right now.

14 29

Add an attribute spec descriptor.
by Eric Snow 21 Jan '14

21 Jan '14

Here's something I've thought about off and on for a while. Occasionally it would be useful to me to have a class attribute I can use to represent an attribute that will exist on *instances* of the class. Properties provide that to an extent, but they are data descriptors which means they will not defer to like-named instance attributes. However, a similar non-data descriptor would fit the bill. For the sake of clarity, here is a simple implementation that demonstrates what I mean. I know it's asking a lot <wink>, but try to focus on the idea rather than the code. I've posted a more complete (and feature-rich) implementation online [1]. class Attr: """A non-data descriptor specifying an instance attribute.""" def __init__(self, name, doc=None): self.__name__ = name self.__doc__ = doc def __get__(self, obj, cls): if obj is None: return self else: # The attribute wasn't found on the instance. raise AttributeError(self.__name__) def attribute(f=None): """A decorator that converts a function into an attribute spec.""" return Attr(f.__name__, f.__doc__) def attrs(names): """A class decorator that adds the requested attribute specs.""" def decorator(cls): for name in names: attr = Attr(name) setattr(cls, name, attr) return cls return decorator Other features not shown here (see [1]): * an optional "default" Attr value * an optional "type" Attr (derived from f.__annotations__['return']) * __qualname__ * auto-setting self.__name__ during the first Attr.__get__() call * a nice repr * Attr.from_func() * proper ABC handling in attrs() (not an obvious implementation) * optionally inheriting docstrings Such a descriptor is particularly useful for at least 2 things: 1. indicating that an abstractproperty is "implemented" on *instances* of a class 2. introspecting (on the class) all the attributes of instances of a class Alternatives: * "just use a property". As already noted, a property would work, but is somewhat cumbersome in the case of writable attributes. A non-data descriptor is a more natural fit in that case. * for #1, "just use a normal class attribute". This would mostly work. However, doing so effectively sets a default value, which you may not want. Furthermore, it may not be clear to readers of the code (or of help()) what the point of the class attr is. Thoughts? -eric [1] https://bitbucket.org/ericsnowcurrently/odds_and_ends/src/default/attribute… [2] Where would Attr/attribute/attrs live in the stdlib? inspect? types?

3 2

Fwd: Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`
by Andrew Barnert 21 Jan '14

21 Jan '14

Sent from a random iPhone Begin forwarded message: > From: Andrew Barnert <abarnert(a)yahoo.com> > Date: January 21, 2014, 4:20:19 PST > To: Ram Rachum <ram.rachum(a)gmail.com> > Cc: "python-ideas(a)googlegroups.com" <python-ideas(a)googlegroups.com> > Subject: Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor` > > On Jan 21, 2014, at 2:17, Ram Rachum <ram.rachum(a)gmail.com> wrote: > >> If you're writing code that needs to use both a lot of IO and a lot of CPU. For example, you're downloading many items from the internet and then doing post-processing on them. > > Yes, but in that case, how could a single executor with n processes and m threads help at all? You can only have one thread per process doing CPU work; they're still going to end up blocking each other. > > And this is very easy to solve: run the downloads on a thread pool, and as each one finishes, kick its post processing off to a process pool. > > But you should be able to build the two-tier pool in under half an hour, and then you can test to find applications where it really does or doesn't help. > >> On Tue, Jan 21, 2014 at 10:42 AM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >>> On Jan 17, 2014, at 5:00, Ram Rachum <ram.rachum(a)gmail.com> wrote: >>> >>> > Hi, >>> > >>> > I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process. >>> >>> What for? >>> >>> Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either. >>> >>> There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor. >>

1 0

Fwd: Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`
by Andrew Barnert 21 Jan '14

21 Jan '14

Google apparently ate this message, and the next one, so... Forwarding them. Apologies for the mess. Apparently you can't just reply to messages that arrive on the list via Google Groups? Sent from a random iPhone Begin forwarded message: > From: Andrew Barnert <abarnert(a)yahoo.com> > Date: January 21, 2014, 0:42:11 PST > To: Ram Rachum <ram.rachum(a)gmail.com> > Cc: "python-ideas(a)googlegroups.com" <python-ideas(a)googlegroups.com> > Subject: Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor` > > On Jan 17, 2014, at 5:00, Ram Rachum <ram.rachum(a)gmail.com> wrote: > >> Hi, >> >> I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process. > > What for? > > Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either. > > There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor.

1 0

Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`
by Andrew Barnert 21 Jan '14

21 Jan '14

On Jan 21, 2014, at 4:20, Andrew Barnert <abarnert(a)yahoo.com> wrote: > And this is very easy to solve: run the downloads on a thread pool, and as each one finishes, kick its post processing off to a process pool. Wait, that's stupid. Even simpler: just use a flat process pool of 2N for everything (or whatever multiplier is appropriate for your load--although often a downloader doesn't want to do more than about 4-12 simultaneous downloads, which is already below 2N on most modern computers...).

1 0

Re: [Python-ideas] Tail recursion elimination
by Haoyi Li 21 Jan '14

21 Jan '14

MacroPy also has an implementation of TCO implemented using trampolining. It trades stack introspection for load-time-analysis, which could be a win or a loss depending on how you view things. ------------------------------ From: Ryan Sent: 1/18/2014 4:57 PM To: musicdenotation(a)gmail.com; Joao S. O. Bueno; python-ideas(a)python.org Subject: Re: [Python-ideas] Tail recursion elimination I wrote one that uses decorators. How is that special syntax? musicdenotation(a)gmail.com wrote: > > On Jan 18, 2014, at 22:08, "Joao S. O. Bueno" <jsbueno(a)python.org.br> > wrote: > > > You can use tail recursion elimination in Python as it is today. > > I have seen many "implementations" of tail-call optimization, and their > common problem is that they all require special syntax to work. I need a > solution that is directly usable with Python's orrdinary *return*statement. > > ------------------------------ > > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

17 24

Re: [Python-ideas] Tail recursion elimination
by Haoyi Li 19 Jan '14

19 Jan '14

> Having to fork the entire compiler just to write a few functions in > their most idiomatic, natural (recursive) form seems a bit extreme, > wouldn't you say? You don't need to. MacroPy's @tco decorator is about as easy as you could ask for. 'pip install macropy', 'from macropy.experimental.tco import macros, tco' is about as easy as you could ask for. Works for arbitrary tail-calls too, not just tail recursion. If you haven't tried it out, complaining about the difficulty of implementing tail-call-optimization yourself seems silly. From: Andrew Barnert Sent: 1/19/2014 12:04 PM To: Steven D'Aprano; python-ideas(a)python.org Subject: Re: [Python-ideas] Tail recursion elimination From: Steven D'Aprano <steve(a)pearwood.info> Sent: Saturday, January 18, 2014 4:45 PM > On Sat, Jan 18, 2014 at 10:29:46AM -0800, Andrew Barnert wrote: > >> Whether or not you really need it, adding it to Python is a fun >> challenge that's worth trying. > > "Need" is a funny thing. Which I why I made that point. It's not a completely objective question, and it may be hard for the OP (or you, or anyone else) to convince anyone that he "needs" it even though he does (or, more importantly, convince people that _they_ need it). If so, he doesn't have to let that stop him from writing and sharing an implementation. It may turn out that, once people have a chance to play with it, that will convince everyone better than any abstract argument he could make. If not, at least he's had fun, learned about CPython internals, and, most importantly, produced a fork that he can maintain as long as he thinks he needs it. Depending on your time and resources, that may not be worth doing, but that's the same decision as any other development project; there's nothing actually stopping anyone from doing it if it's worth their while, so anyone who wants this should consider whether it's worth their while to do it. > You can go a long way without recursion, or only shallow recursion. In > 15 years + of writing Python code, I've never been in a position that I > couldn't solve a problem because of the lack of tail recursion. But > every time I manually convert a recursive algorithm to an iterative one, > I feel that I'm doing make-work, manually doing something which the > compiler is much better at than I am, and the result is often less > natural, or even awkward. (Trampolines? Ewww.) But the same is true for converting a naive recursive algorithm to tail-recursive. It's unpleasant make-work, just like converting it to iteration. In a language like Common Lisp, it's about the same amount of work, but the tail-recursive version often ends up looking more natural. In a language like Python, where we typically deal in iterables rather than recursive data structures, I believe it would often be _more_ work rather than the same amount, and end up looking a lot less natural rather than more. I'm sure there would be exceptions, but I suspect they would be rare. >> Third, eliminating tail calls means the aren't on the stack at >> runtime, which means there's no obvious way to display useful >> tracebacks. I don't think too many Python users would accept the >> tradeoff of giving up good tracebacks to enable certain kinds of >> non-pythonic code, > > What makes you say that it is "non-pythonic"? You seem to be assuming > that *by definition* anything written recursively is non-pythonic. Not at all. There's plenty of code that's naturally recursive even in Python—and much of that code is written recursively today. For a good example, see os.walk. However, the main driver for TCE is the ability to write looping constructs recursively, which is not possible without it (unless the thing you're looping over is guaranteed not to be too big). Look at any tutorial on tail recursion; it's always recursing over a cons list or something similar. And looping that way in Python will almost always be non-pythonic, because you will have to drive the iterable manually. Again, there are surely exceptions, but I doubt they'd be very common. > In fact, in some cases, I *would* willingly give up *non-useful* > tracebacks for the ability to write more idiomatic code. Have you seen > the typical recursive traceback? But if you eliminate tail calls, you're not just eliminating recursive tracebacks; you're eliminating every stack frame that ends in a tail call. Which includes a huge number of useful frames. If you restrict it to _only_ eliminating recursive tail calls, then it goes from something that can be done at compile time (as I showed in my previous email) to something that has to be done at runtime, making every function call slower. And it doesn't work with mutual or indirect recursion (unless you want to walk the whole stack to see if the function being called exists higher up—which makes it even slower, and also gets us back to eliminating useful tracebacks). > py> a(7) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "./rectest.py", line 2, in a > return b(n-1) > File "./rectest.py", line 5, in b > return c(n-1) + a(n) > File "./rectest.py", line 2, in a > return b(n-1) > File "./rectest.py", line 5, in b > return c(n-1) + a(n) > File "./rectest.py", line 2, in a > return b(n-1) > File "./rectest.py", line 5, in b > return c(n-1) + a(n) > File "./rectest.py", line 2, in a > return b(n-1) > File "./rectest.py", line 5, in b > return c(n-1) + a(n) > File "./rectest.py", line 2, in a > return b(n-1) > File "./rectest.py", line 5, in b > return c(n-1) + a(n) > File "./rectest.py", line 2, in a > return b(n-1) > File "./rectest.py", line 5, in b > return c(n-1) + a(n) > File "./rectest.py", line 9, in c > return 1/n > ZeroDivisionError: division by zero > > The only thing that I care about is the very last line, that function c > tries to divide by zero. The rest of the traceback is just noise, I > don't even look at it. Your example is not actually tail-recursive. I'm guessing you know this, and decided that having something that blows up fast just to have an example of a recursive traceback was more important than having an example that also fits into the rest of the discussion—which is perfectly reasonable. But it's still worth calling that out, because at least half the blog posts out there that say "Python sucks because it doesn't have TCE" prove Python's suckiness by showing a non-tail-recursive algorithm that would blow up exactly the same way in Scheme as in Python. > Now, it's okay if you disagree, or if you can see something useful in > the traceback other than the last entry. Sure. Unless that line in b is the only place in your code that ever calls c, I think it would be useful to know how we got to c and why n is 0. If that isn't useful, than _no_ tracebacks are ever useful, not just recursive ones. > I'm not suggesting that TCE > should be compulsary. I would be happy with a commandline switch to > turn it on, or better still, a decorator to apply it to certain > functions and not others. I expect that I'd have TCE turned off for > debugging. But the primary reason people want TCE is to be able to write functions that otherwise wouldn't run. Nobody asks for TCE because they're concerned about 2KB wasted on stack traces in their shallow algorithm; they ask for it because their deep algorithm fails with a recursion error. So, turning it off to debug it means turning off the ability to reproduce the error you're trying to debug. >> but even if you don't solve this, you can always >> maintain a fork the same way that Stackless has been doing. > > Having to fork the entire compiler just to write a few functions in > their most idiomatic, natural (recursive) form seems a bit extreme, > wouldn't you say? Not necessarily. The whole reason Stackless exists is to be able to write some algorithms in a natural way that wasn't possible with mainline CPython. At least early on, it looked at least plausible that Stackless would eventually be merged into the main core, although that turned out not to happen. There are some core language changes that were inspired by Stackless. Someone (Ralf Schmidt, I think?) was able to extract some of Stackless's functionality into a module that works with CPython, which is very cool. But even without any of that, people were able to use Stackless when they wanted to write code that required its features. That's surely better than not being able to write it, period. And a TCE fork could go the same way. It might get merged into the core one day, or it might inspire some changes in the core, or it might turn out to be possible to extract the key functionality into a module for CPython—but even if none of that happens, you, and others, can still use your fork when you want to. If you prefer to call it a patch or a branch or something else instead of a fork, that's fine, but it's basically the same amount of work either way, and there's nothing stopping anyone who wants it from doing it. _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Make print() not append line break by default
by musicdenotation＠gmail.com 19 Jan '14

19 Jan '14

And add println()

4 3

Re: [Python-ideas] Create Python 2.8 as a transition step to Python 3.x
by Andrew Barnert 19 Jan '14

19 Jan '14

On Jan 18, 2014, at 17:12, Neil Schemenauer <neil(a)python.ca> wrote: > On 2014-01-17, Andrew Barnert wrote: >> What exactly do you mean by "the bytes/unicode changes"? > > I mean all of those things that you listed. bytes = the Python 2.7 > str object, str object is Python 2.7 unicode object. If you really mean I the things I listed, we already have that. It's called Python 3.4. If you want to fork it and rename it 2.8, I can't imagine who that would help. A smaller list of changes _might_ mean a useful intermediate target, but if you're not even willing to think through the issues and discuss them, you're not going to come up with such a list. >> I suspect that, whatever your exact answers, it would be a lot >> easier to fork 3.4 and port the 2.7 behavior you want than to fork >> 2.7 and backport almost all of 3.4. > > It's a lot of work no matter which way you do it. That's one of the > biggest problems with this idea. > >> And if you do it that way, you could even adapt the idea someone >> proposed a few weeks ago—not popular on this list, but maybe >> popular with your target audience—of turning each change on and >> off with a "from __past__ import misfeature" statement, so people >> could pick and choose the ones they need, and gradually remove >> past statements as they port from your forked 2.8 to real 3.4. > > You can't make those changes with __future__/__past__ imports. They > effect the whole runtime, not single module. Sure you can. It already works for __future__ Unicode literals in 2.7. Most of the other changes would work just as well. A few might not--but again, you have to go through them one by one and decide. >> However, I also suspect that, whatever your exact answers, it >> won't be that useful. Look at people's reasons for not moving to >> 3.x: > > Imagine I'm a developer with the Python 2.x codebase. I'm either > lazy or I'm too busy with other company projects that I can't put > the effort into porting my 2.x code to 3.x, even if all the 3rd > party libraries have been ported. > > How can we make it easier for them to move their code towards Python > 3.x rather than keeping it as 2.x? Not by publishing something that requires the exact same code changes as 3.4 and calling it 2.8. That might trick a handful of devs, and help a handful of others trick their managers, but that's not much benefit. > A maintained interpreter to run > Python 2.x code is going to continue to exist. Some python-dev > people seem to suggest we can suggest that end of maintenance of > Python 2.7 is going to force people to port their code. That's > ridiculous. I've never heard anyone suggest this. The people who are most gung ho about 3.x are the ones who keep pointing out that many apps never need to port and that people like RedHat are likely to continue supporting 2.7 long after the PSF stops doing so. > I want to make it more attractive for these developers to move > towards Python 3 rather than stalling out on Python 2.7 forever. > How best to do that is still to be determined. I think my 2.8 idea > might be better than the status quo but it's just a crazy idea. A crazy idea is one thing; a misinformed idea is another. > >> I'm having a hard time imagining code that would be easy to port to 2.8, but not to 3.x. For example: >> >> payload = <some object with a __str__ method to serialize it> >> sock.sendall('Header: {}\r\nAnother: {}\r\n\r\n{}'.format( >> headers['header'], headers['another'], payload)) >> >> Even with just the two changes you already suggested: First, you >> have to change the literal to a bytes literal. > > That part is easy, could even be done with an automated tool > (change u' to ' and ' to b'). > >> More seriously, you have to rename that payload type's __str__ >> method to __bytes__. > > Nope, no __bytes__ in my proposed 2.8. Then the code just doesn't work. The payload types existing __str__ returns a bytes object, which raises a TypeError. >> And if it does any string stuff internally, like encoding JSON, >> that has to change. Meanwhile, your logging code probably relies >> on the same _str__ method actually returning a str, so you have to >> add one of those. Assuming headers is a dict of strs, you either >> need to go back up the chain (or into the API that provides it) >> and change that so it's been a dict of bytes all along, or you >> need to explicitly encode the headers here. That doesn't sound too >> hard overall… but that gives you working Python 3.5 code (assuming >> PEP 460 goes through). And there doesn't seem to be any shortcut >> that would give you working 2.8 code without also working in 3.5. > > I think you are misunderstanding my proposal, no problems like the > ones you suggest, bytes() would be the Python 2.7 str class. All > the internal bytes/unicode internals would be like 2.7. You're contradicting yourself. You explicitly said that your proposal includes all of the changes I suggested. That includes, right near the very top, things like no automatic str/bytes conversions. But you seem to be assuming they would still exist even though you decided to remove them. > That's > basically the whole idea of this proposal, the bytes/str change in > 3.x is the really disruptive one, separate it into separate > interpreter versions But you've proposed that the all of the elements of the str/bytes change should be part of 2.8, which means it will be just as disruptive as 3.4.

1 0