Mailman 3 October 2005 - Python-Dev

Re: [Python-Dev] Pythonic concurrency - cooperative MT
by Martin Blais Oct. 13, 2005

Oct. 13, 2005

On 10/1/05, Antoine <solipsis(a)pitrou.net> wrote: > > > like this with their "deferred objects", no? I figure they would > > need to do something like this too. I will have to check.) > > A Deferred object is just the abstraction of a callback - or, rather, two > callbacks: one for success and one for failure. Twisted is architected > around an event loop, which calls your code back when a registered event > happens (for example when an operation is finished, or when some data > arrives on the wire). Compared to generators, it is a different way of > expressing cooperative multi-threading. So, the question is, in Twisted, if I want to defer on an operation that is going to block, say I'm making a call to run a database query that I'm expecting will take much time, and want to yield ("defer") for other events to be processed while the query is executed, how do I do that? As far as I remember the Twisted docs I read a long time ago did not provide a solution for that.

25 51

Re: [Python-Dev] Extending tuple unpacking
by Delaney, Timothy (Tim) Oct. 13, 2005

Oct. 13, 2005

Paul Du Bois wrote: > On 10/10/05, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> cmd, *args = input.split() > > These examples also have a reasonable implementation using list.pop(), > albeit one that requires more typing. On the plus side, it does not > violate > DRY and is explicit about the error cases. > > args = input.split() > try: > cmd = input.pop(0) > except IndexError: > cmd = '' I'd say you violated it right there ... (should have been):: args = input.split() try: cmd = arg.pop() except IndexError: cmd = '' FWIW, I've been +1 on * unpacking since I first saw the proposal, and have yet to see a convincing argument against it other than people wanting to stick the * anywhere but at the end. Perhaps I'll take the stdlib challenge (unfortunately, I have to travel this weekend, but I'll see if I can make time). Tim Delaney

9 18

Re: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
by john.m.camara＠comcast.net Oct. 13, 2005

Oct. 13, 2005

> > Guido van Rossum writes: > > Code that *doesn't* need Queue but does use threading > > shouldn't have to pay for loading Queue.py. > > Greg Ewing responds: > > What we want in this kind of situation is some sort > > of autoloading mechanism, so you can import something > > from a module and have it trigger the loading of another > > module behind the scenes to provide it. > > John Camera comments: > > Bad idea unless it is tied to a namespace. So that users knows > > where this auto-loaded functionality is coming from. Otherwise > > it's just as bad as 'from xxx import *'. > > Michael Chermside comments: > John, I think what Greg is suggesting is that we include Queue > in the threading module, but that we use a Clever Trick(TM) to > address Guido's point by not actually loading the Queue code > until the first time (if ever) that it is used. > > I'm not familiar with the clever trick Greg is proposing, but I > do agree that _IF_ everything else were equal, then Queue seems > to belong in the threading module. My biggest reason is that I > think anyone who is new to threading probably shouldn't use any > communication mechanism OTHER than Queue or something similar > which has been carefully designed by someone knowlegable. > I guess from Gregs comments Im not sure if he wants to import threading and as a result Queue becomes available in the local namespace and bound/loaded when it is first needed and thus saves himself from typing import Queue immediately after import threading or Queue becomes part of the threading namespace and bound/loaded when it is first needed. Queue then becomes accessible through threading.Queue When Greg says > However, it does seem awkward to have a whole module > providing just one small class that logically is so > closely related to other threading facilities. It sounds like he feels Queue should just be part of threading but queues can be used in other contexts besides threading. So having separate modules is a good thing. The idea of delaying an import until its needed sounds like a great idea and having built in support for this would be great. Here are 2 possible suggestions for the import statements import Queue asneeded delayedimport Queue # can't think of a better name at this time But auto loading a module by a module on behalf of a client just doesnt sit too well for me. How about the confusion it would cause. Is Queue in treading module a reference to a Queue in a Queue module or a new class all together? If we go down this slippery slope we will see modules like array, struct, etc getting referenced and getting auto loaded on behalf of the client. Where will it end. John M. Camara > > Guido van Rossum writes: > > Code that *doesn't* need Queue but does use threading > > shouldn't have to pay for loading Queue.py. > > Greg Ewing responds: > > What we want in this kind of situation is some sort > > of autoloading mechanism, so you can import something > > from a module and have it trigger the loading of another > > module behind the scenes to provide it. > > John Camera comments: > > Bad idea unless it is tied to a namespace. So that users knows > > where this auto-loaded functionality is coming from. Otherwise > > it's just as bad as 'from xxx import *'. > > John, I think what Greg is suggesting is that we include Queue > in the threading module, but that we use a Clever Trick(TM) to > address Guido's point by not actually loading the Queue code > until the first time (if ever) that it is used. > > I'm not familiar with the clever trick Greg is proposing, but I > do agree that _IF_ everything else were equal, then Queue seems > to belong in the threading module. My biggest reason is that I > think anyone who is new to threading probably shouldn't use any > communication mechanism OTHER than Queue or something similar > which has been carefully designed by someone knowlegable. > > -- Michael Chermside >

3 2

Re: [Python-Dev] Extending tuple unpacking
by Michael Chermside Oct. 13, 2005

Oct. 13, 2005

Steve Holden writes: > I do feel that for Python 3 it might be better to make a clean > separation between keywords and positionals: in other words, of the > function definition specifies a keyword argument then a keyword must be > used to present it. > > This would allow users to provide an arbitrary number of positionals > rather than having them become keyword arguments. At present it's > difficult to specify that. Antoine Pitrou already responded: > Do you mean it would also be forbidden to invoke a "positional" argument > using its keyword? It would be a huge step back in usability IMO. Some > people like invoking by position (because it's shorter) and some others > prefer invoking by keyword (because it's more explicit). Why should the > implementer of the API have to make a choice for the user of the API ? I strongly agree with Antoine here, but the combination of "keyword arguments after the star": > foo(a, b, c=1, *args, d=2, e=5, **kwargs) > ^^^^^^^^^ ^^^^^^^^ > positional only with kw > or with kw with "star without a name": > def f(a, b, c=default, *, foo=1, bar=2): pass > > Here, c is a positional argument with a default value, while foo and bar are > forced to be keyword arguments. is quite tempting. It satisfies Steve by allowing the implementer of the function to require keyword arguments. It satisfies Antoine and myself by also allowing the implementor of the function to permit positional OR keyword use, and making this the default behavior. It is logically consistant. There's just one big problem that I know of: Guido writes: > I've always wanted to write that as > > f(a, b, *args, foo=1, bar=2, **kwds) > > but the current grammar doesn't allow it. Hmm.... why doesn't the current grammar allow it, and can we fix that? I don't see that it's a limitation of the one-token-lookahead, could we permit this syntax by rearanging bits of the grammer? -- Michael Chermside

2 1

Re: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
by Michael Chermside Oct. 12, 2005

Oct. 12, 2005

Aahz writes: > I'm suggesting that we add a doc note that using the thread module is > discouraged and that it will be renamed in 3.0. Then we're apparently all in agreement. -- Michael Chermside

1 0

Re: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
by Michael Chermside Oct. 12, 2005

Oct. 12, 2005

Aahz writes: > (Python 3.0 should "deprecate" ``thread`` by renaming it to ``_thread``). Guido says: > +1. (We could even start doing this before 3.0.) Before 3.0, let's deprecate it by listing it in the Deprecated modules section within the documentation... no need to gratuitously break code by renaming it until 3.0 arrives. -- Michael Chermside

3 3

Re: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
by john.m.camara＠comcast.net Oct. 12, 2005

Oct. 12, 2005

> Skip write: > Is the Queue class very useful outside a multithreaded context? The notion > of a queue as a data structure has meaning outside of threaded applications. > Its presence might seduce a new programmer into thinking it is subtly > different than it really is. A cursory test suggests that it works, though > q.get() on a empty queue seems a bit counterproductive. Also, Queue objects > are probably quite a bit less efficient than lists. Taken as a whole, > perhaps a stronger attachment with the threading module isn't such a bad > idea. > Maybe Queue belongs in a module called synchronize to avoid any confusions. John M. Camara

2 1

Active Objects in Python
by Bruce Eckel Oct. 12, 2005

Oct. 12, 2005

According to this list's welcome message, I should introduce myself. I'm Bruce Eckel, I write, consult and give seminars about computer programming, I use Python whenever I can get away with it, and I've spoken at Pycon a number of times. There are further URLs in my signature at the bottom. I'm joining this list because I was having a conversation with Guido about concurrency support in Python (in particular, using an actor/active-object approach) and he suggested I bring it over here. Here are the highlights of my most recent reply to him (Guido's remarks are marked by the '>' signs): > Agreed. IMO the pthreads-style solution doesn't work well no matter > *what* the programming model. Exactly. I think the problem may be that low-level threads people are one step behind (much like embedded systems programmers, which is where I began). They may be just now catching up to objects, but as far as concurrency goes their brains still think in terms of threads, so it seems natural to apply thread concepts to objects. But from an OO standpoint, pthread-style thinking is two steps backwards. You effectively throw open the innards of the object that you just spent time decoupling from the rest of your system, and the coupling is now unpredictable. It's the worst of all possible worlds. > I worked on Mobile Agents at CNRI from 1995 till 2000, but they were > very heavy-weight. Yet, I think that any solution that moves objects > between CPUs on a regular basis is going to suffer the same problem > that process switching in general gets you -- the move is so expensive > that it's hard to decide when it's justified. The examples I've seen haven't relied on mobility, so that's a question: how important is mobility really to an agent system? And could it be done in a trivial fashion in Python, by sending source code. And if it was really necessary sometimes, could it be something where the cost of mobility only occurred when moving an object? It's possible that either an argument can be made against mobility and/or some kind of trivial mobility system could be developed that would work when necessary. Possibly a Linda-like put-take system with pickles (off the top of my head). I don't know that much about mobility, but it does seem like mobile agents could be a powerful solution to the problem solved by enterprise messaging systems. > I believe you pointed me to Active Objects before, right? (Years ago, > maybe when you visited us at Zope.) I may have mentioned them in the past. > If multiple active objects can co-exist in the same *process*, but > nevertheless the language implementation prevents them from sharing > data except via channels, and in addition allows dynamic reallocation > of active objects across multiple CPUs, they just might be the ticket. > But it would require a complete implementation to prove it. Yes, defining an class as "active" would: 1) Install a worker thread and concurrent queue in each object of that class. 2) Automatically turn method calls into tasks and enqueue them 3) Prevent any other interaction other than enqueued messages At that point, the only time the GIL might be needed is to lock the queue at the point of putting ant taking objects. But there has been work done on lock-free data structures, which has been incorporated into Java 5 libraries, so it might even be possible to use a lock-free queue to do this and thus eliminate the need for the GIL at all. Since the enqueuing process serializes all requests to active objects, and since each message-task runs to completion before the next one starts, the problem of threads interfering with each other is eliminated. Also, the enqueuing process will always happen, so even if the queue blocks it will be for a very short, deterministic time. So the theory is that Active Object systems cannot deadlock (although I believe they can livelock). So yes, the best way for this to work might be some kind of enforced implementation -- but forcing you to do something is not particularly Pythonic, so I would think that it would be possible to build an active object framework along with some checking tools to warn you if you are breaking the rules. But what's not clear is whether this would require knowledge of the innards of the Python interpreter (which I don't have) or if it could be built using libraries. BTW: I think that Hoare's Communicating Sequential Processes (CSP) basically describes the idea of active objects but without objects. That is, I think active objects is CSP applied to OO. Bruce Eckel http://www.BruceEckel.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel

16 24

Europeans attention please!
by Guido van Rossum Oct. 12, 2005

Oct. 12, 2005

I have some 65%-off passes to EuroOSCON which starts next Monday in Amsterdam. Anybody interested? http://conferences.oreillynet.com/eurooscon/grid/ -- --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
by john.m.camara＠comcast.net Oct. 12, 2005

Oct. 12, 2005

Greg Ewing wrote: > > Guido van Rossum wrote: > > > I see no need. Code that *doesn't* need Queue but does use threading > > shouldn't have to pay for loading Queue.py. > > However, it does seem awkward to have a whole module > providing just one small class that logically is so > closely related to other threading facilities. > > What we want in this kind of situation is some sort > of autoloading mechanism, so you can import something > from a module and have it trigger the loading of another > module behind the scenes to provide it. > Bad idea unless it is tied to a namespace. So that users knows where this auto-loaded functionality is coming from. Otherwise it's just as bad as 'from xxx import *'. John M. Camara

1 0