Mailman 3 May 2009 - Twisted

[Twisted-Python] My twisted web server fall down
by Muaaz Hussain May 24, 2009

May 24, 2009

Dear all, I ran penetration testing tool against my twisted web server , the tool used to make a huge number of threads, all that threads make a connection to the twisted concurrently. when I made the number of threads more than 1024 the twisted web server eventually has fall down. is there any method to let the twisted web server to limit the concurrent connection per second or per host.

5 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com May 24, 2009

May 24, 2009

1 0

[Twisted-Python] Available for short-term Twisted consulting, e.g. training, design (but not coding)
by Itamar Shtull-Trauring May 22, 2009

May 22, 2009

Need help with Twisted, distributed systems or or just plain old software architecture? Due to a repetitive strain injury I am unable to type much (though I am slowly improving), and so I'm taking time off of work to recover. As a result I am available for short-term consulting until I recover, e.g. training or design work, assuming of course it does not involve any typing. I've worked on Twisted itself, as well as projects ranging from high-performance distributed C++ servers, to an access … [View More]

2 1

[Twisted-Python] Twisted training
by Thomas Vander Stichele May 22, 2009

May 22, 2009

Hi people, Is there anyone in the community who is available for a Twisted training job ? Our team of developers is growing, and we are considering our options on how we can smooth out their learning cycle. Getting a focused Twisted training is definately something we're considering. The training would be on-site in Barcelona, Europe, if that helps you make your decision :) Feel free to either reply publically or privately. Thanks Thomas -- I won't leave you all you have is that spell … [View More]

1 0

Re: [Twisted-Python] Consistent interfaces to asynchronous partially-available services using Deferreds and state machines (was Re: Another approach to allowing __init__ to work with Deferreds)
by Terry Jones May 21, 2009

May 21, 2009

Hi Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: > On 11 May, 04:19 pm, terry.jones(a)gmail.com wrote: > > >I posted to this list back in Nov 2008 with subject: > >A Python metaclass for Twisted allowing __init__ to return a Deferred > > Let me try rephrasing your use-case here, for two reasons: one, I want to > make sure I fully understand it, and two, I feel like the language this is > couched in (hacks about __init__ and … [View More]Deferreds and > metaclass/mixin/decorator [ab]use) are detracting from the real core > use-case here. OK, fair enough. I'm happy to work upwards from my concrete problems to a solution to a more general problem. > You have a utility object which you want to create immediately and make > immediately available to various pieces of calling code. However, an > instance of this class represents a shared interface to an external, > asynchronous resource, to which you must establish a connection, and so > you don't immediately have a connection when the class is created. Yes, though I don't know why you use the word "shared". Perhaps because a created instance might be passed to several other pieces of code that all use it? > However, you want to contain all this complexity behind a nice facade, > and tell all the callers "just call these (Deferred-returning) methods > and you will get sensible results no matter what state the connection is > in". Yes. And as I mentioned in my later mail to Drew, a nice property of keeping the complexity behind the interface is that when a vanilla class that is being used in the manner you describe, is changed from being reliably in a single state to having multiple states, the caller is not aware of that and does not have to change how it instantiates or uses instances of the class. > If my assessment of your use-case is flawed, please say so Nope, that's about perfect. > To skip ahead to the end: the answer is that you want a state-machine. > And it is quite sad to me that Twisted doesn't have a nice, standard, > full-featured state-machine class that we use for everything like this, > because members of the Twisted team have implemented at least half a > dozen of these, probably a lot more, in various applications. I am like > 90% sure that there's a ticket in the tracker for this, but I couldn't > find it by searching around a bit. I hope exarkun or jml or radix will > have a better memory of this than I do. I'll be interested to hear. I've omitted a big chunk of your reply here - that I agree with and which recaps what went down in the original thread and comments on Drew's suggestion. > But, although I still think this is generally good practice, it doesn't > solve the underlying problem I think you're really getting at: > consistency and convenience in the face of Deferred-ness. I still like your approach (using a class method to hand you a fully initialized instance), but didn't find it appropriate for my situation. The main problem was that I was writing code in an __init__ method of a class that was already in use by other code, including being called from the __init__ method of other classes. Your solution is fine if you're in a context where you can properly deal with deferreds. If you're not (e.g., you're in an __init__ method) then calling something that creates you an instance of another class via a deferred just leaves you with the same problem. I hope that's making sense. Is the underlying problem "consistency and convenience in the face of Deferred-ness"? You could look at it that way (and I'm happy to). My __init__ case seems to be well summarized by my original comment: this is a general problem of the synchronous world (in which __init__ is supposed to prepare a fully-fledged class instance and cannot return a deferred) meeting the asynchronous world in which we would like to (and must) use deferreds. If Python allowed me to return a deferred from __init__, my problem would vanish. That's not going to happen though, I know :-) Yours would remain though, and as you say, it's more general. > Applications have to handle Deferreds from the connection's methods > anyway, and there's no reason to force them to all have code to handle at > least two (one for the connection, one for the actual application-level > message), where one would do fine. I'm not 100% sure that I follow this, but I think so. > >Anyway.... fast forward 6 months and I've hit the same problem again. > >It's with existing code, in which I would like an __init__ to call > >something that (now, due to changes elsewhere) returns a deferred. So I > >started thinking again, and came up with a much cleaner way to do the > >alternate approach via a class mixin: > > I think I like this a bit better than your earlier approaches. It's > automatic, its semantics are pretty clear, and it doesn't require any abuse > of __init__'s implicit contract; your instance *is* in a fully valid state > when it's created, it's just a different state than the state that it's in > later. However, you can still call all the same methods and get the same > results. Yes, those are the advantages. And the different state will, in most use cases (I claim), be quite fleeting. It's that short-term not-quite-ready window that the temporary state accounts for. And if the not-quite-ready happens to not be short, then providing functionality like this (not necessarily my implementation) is even more important. I mention all this, for clarity, not for you - I know you already know - but for others who might be reading along now or later. > It still has one major flaw given your earlier example of a database > connection (as I described above): it doesn't handle errors very well. Right. > In particular - and this is why you really need a state machine - it > doesn't handle the case where errors start happening *later*. OK, more on this below. > It's also got a few implementation issues that you might not be aware of > though - and you seem to appreciate a lot of detail in these responses, > so I'll just look at it line by line, code-review style. Yes, that's great, and thanks. > I apologize in advance if this sounds like I'm being hypercritical - I > realize you may have omitted certain details to keep this brief for > discussion and so may have been aware of most of these problems. Again, > even if you fully understood all of these details I am sure there are > many readers who didn't though :) Some I'm aware of and skipped, others not. I never know how much detail to provide / go into, or if I'm bugging people on the list, etc. > > from twisted.internet import defer > > > > class deferredInitMixin(object): > > def wrap(self, d, *wrappedMethods): > > Just as a point of convenience, I would have automatically determined this > list of method names by using a decorator or something. Having it as a > static list in the method invocation seems to me like it would be very easy > to forget to add or remove a method from the list, and it would make diffs > that touched a user of this class always have two hunks for adding a > method; one at the method definition site, one at the call to wrap(). I started out trying to write this using decorators. But I didn't really see how to do it. I was using two - one for __init__ and one for the wrapped functions. I also tried with decorators and a super class. In the end I saw a simple way to do it with the mixin, so went for that. I'd be happier with a decorator solution for the reasons you mention. > Also, it's not really clear to me how cooperative invocations of wrap() are > meant to work with inheritance. Using a decorator on methods which were > intended to be deferred wouldn't fully solve that problem (you've still got > to sort out what order methods get restored in, or if there are multiple > calls to wrap() in different places in the inheritance tree which methods > go with which Deferreds) but it would at least provide a convenient > starting place to put that information. Yes.... > > self.waiting = [] > > self.stored = {} > > I'd make these attributes private if I were you. I am pretty sure that > you don't ever want application code poking around in there :). Right. A bad habit of mine. I did at least think of this afterwards :-) > > def restore(_): > > for method in self.stored: > > setattr(self, method, self.stored[method]) > > The reference you're cleaning up here has some edge-cases. For example, > if some other code comes along and grabs what it thinks is a regular > bound method from your instance, and then invokes it after the Deferred > has completed, it will still have the original method. You mean it will still have the wrapped method, I think. Agreed that's a problem. I was originally going to look at d.called in the mixin class to short-circuit the wrapped behavior if the deferred had fired. I should do something like that, else the deferred from the wrapper will never fire - which is what I think you're saying. > There are also some less severe, but potentially very confusing issues > with making every instance of your class always participate in a > bazillion circular references. I don't think I fully understand this. The instance of my class only has wrapped functions for a (typically?) very short time. I don't see the circular references, but OTOH I haven't thought about that at all... > By itself, this isn't really worth worrying about (Python added a garbage > collector for a reason, after all) but it has historically been > problematic in areas like making debugging memory leaks tricky. > Especially when the circular references run through stack frames which > refer to Deferreds :). So if you do dynamically replace a method on a > class, it's better to clean it up with delattr() than a subsequent > setattr(). You mean "then" a subsequent setattr, right? And thanks, I didn't know that at all. > This wrapper doesn't preserve function metadata Right - that was something I deliberately left out. I've even used t.p.u.mergeFunctionMetadata in the past :-) > I think some other decorator libraries have cuter / easier to use > implementations of the same thing, this problem is not unique to > Twisted). There's also functools.update_wrapper > > d.addCallback(restore) > > Here, on the final line, we come to the more serious problem of this > approach: there's no error handling. If the underlying Deferred > encounters an errback, then all methods of this class will forever return > Deferreds that never fire. Ah yes :-) Side note: I recently escaped from a fundamentalist religious organization amongst whose axioms are "There Are No Accidents", and "Everything Happens For A Reason". Accordingly, they code in a version of Python that doesn't even have exceptions, and use a fork of Twisted in which deferreds don't have an errback chain. Ahem. You're right. At the very least I should add an errback that errbacks the waiting calls, probably restores the methods, and returns the failure. That's not a full solution, but it's better. > Of course you could chalk up a failed connect Deferred to a failed > startup and just reboot the process, but that pollutes your callers with > knowledge of whether they're calling methods during startup. Yes. There's also the question of what failure to pass to them, suppose I do errback them (which I think I should). > More importantly and realistically though - there's something that > happens *later* which is never covered. What happens when we *lose* the > connection to the database? Assuming a sensible underlying interface, > everybody starts getting errbacked Deferreds, but in most systems like > this you want some recovery facility. And then you're not talking about > just interesting behavior of __init__, but potentially of every method on > the entire class. Yes, and now we're into more interesting territory, where your state machine solution would be nice to have. > As I mentioned above, we've implemented this mechanism in other projects. > One of them is Axiom. Axiom has a batch-processing service which is a > process pool that starts on demand, and tries to present a consistent > interface to its callers regardless of what state the actual processes are > in. (This was written in no small part because we were using libraries > which were flaky and unreliable and wanted to isolate their usage behind a > nice facade which wouldn't freak out if they segfaulted.) > > You can see a usage of our library here, which I believe meshes with your > use-case: > > http://divmod.org/trac/browser/trunk/Axiom/axiom/batch.py?rev=15165#L709 OK, I'll go check this out. I'd have done it already but commenting on it here would make this reply even longer. > I was originally going to include it inline here, but it turned out to be > >100 lines of code to get the whole idea across, so I put it up here: > > http://divmod.org/trac/browser/sandbox/glyph/modality.py?rev=17275 > > This is still missing a lot of details, like for example handling truly > failed connections (i.e. invalid credentials), timeouts and backoff, > redirects, etc. Still, I hope it's somewhat obvious how you would add > additional methods beyond "bork()" to that example. Yes. It looks clean and nice. But I'll have to spend time reading it again and thinking about it to say more. > It would be possible, I think, to implement a layer on top of epsilon.modal > which would provide this pattern exactly so that you just need to plug in > your retransmission and connection rules rather than doing it for every > different application and protocol; that would be really cool. Yeah, I'd use it :-) > epsilon.modal is missing a few useful features, and has a few bugs. I'm > hoping that by drawing attention to it we can get some contributions from > people who are enthusiastic about abstractions like this (hi, Terry! ;-)) Glyph, hi! > >I quite like this approach. (...) It's nice because you don't reply with > >an error and there's no need for locking or other form of coordination - > >the work you need done is already in progress, so you get back a fresh > >deferred and everything goes swimmingly. > > IMHO this is a very important property. The high-level abstract API should > really have fewer failure modes and differing states for its callers to > know about than the lower-level one - really that's the whole point :-). Yes, agreed. > >Comments welcome / wanted. > > Enough comments for you? ;-) Yep, and thanks for taking so much time and going into detail. I'll no doubt continue to think about this. And I'll go look at epsilon.modal. Terry [View Less]

3 4

[Twisted-Python] First time install of Twisted fails
by Uncle Roastie May 20, 2009

May 20, 2009

I'm trying to do an install of Twisted for Python 2.5.4 on Windows XP. I installed PyCrypto, OpenSSL for Windows, and PyOpenSLL. When I enter the Python interpreter and say >>> import OpenSLL I get the following error: "The procedure entry point SSL_CTX_set_info_callback could not be located in the dynamic link library ssleay32.dll." The DLL lives inside the OpenSLL bin directory, which is also on the search PATH. Any ideas? Uncle Roastie roastie(a)rochester.rr.com

2 1

[Twisted-Python] XML-RPC for paralell clients
by Pet May 19, 2009

May 19, 2009

Hi, after playing with reactor, factory and protocol, I'm trying to implement XML-RPC server, but it seems that it is not able to handle several clients in parallel. If I call the slow function with first client and then with second client a fast function, second client will receiver the results after slow function returned. There is no factory for clients in server.Site() like in Protokoll? What is the way to do that for XML-RPC? Thanks for your help! Pet from twisted.web import xmlrpc, … [View More]

2 3

[Twisted-Python] Epsilon installation?
by Jasper May 19, 2009

May 19, 2009

I'm interested in taking a look at divmod Vertex, which requires divmod Epsilon, which I downloaded from here: http://divmod.org/trac/wiki/DivmodEpsilon However, when I try to 'setup.py install' Epsilon, I get the following error: Traceback (most recent call last): File "C:\Documents and Settings\Jasper\Desktop\Epsilon-0.5.12\setup.py", line 6, in ? version=version.short(), File "C:\Python24\lib\site-packages\twisted\python\versions.py", line 37, in short svnver = self.… [View More]

1 1

[Twisted-Python] File persistence without blocking the main event loop?
by Arun Thampi May 18, 2009

May 18, 2009

Hi guys - I'm just beginning to learn about Twisted and was wondering if there is a way to use the built-in Twisted persistence libraries such as dirdbm etc. without blocking the main event loop? I read about an approach which uses deferToThread (comment#3 here: http://code.activestate.com/recipes/413609/) but if you are persisting quite often, doesn't the creation of threads on every persist-call make it very expensive? Any pointers will be much appreciated. Thanks guys! Cheers, Arun -- It'… [View More]

3 3

[Twisted-Python] help using deferred
by Luigi Conte May 18, 2009

May 18, 2009

Hi people, I'm luigi and I'm a student at Politecnico di Milano, Italy. I'm working for a graduate thesis on a framework called usher. It provides an API to do connect e command operations. It manages all those calls with a defferred object (i think). In fact when I call the connect it returns me a defferred object. All that i want to know is how to call the other operations with this object and how to know when an operation is completely done. The procedure I call is simple: connect method, … [View More]start method many times, stop method, disconnect method. I want to call these methods sequentially because I have to fallow a list of commands so I have to be sure the previous methods are already been invoked. I'll post the main methods hoping someone could help me in how to add the method in the deferred and how to know when it is completely invoked. # Connect def connect(self, ctrl_host, ctrl_port, user, passwd): """Connect to an Usher control server. @param ctrl_host: fqdn where Usher controller is running @type ctrl_host: string @param ctrl_port: port number Usher controller is listening on @type ctrl_port: int @param user: username of user connecting to Usher controller @type user: string @param passwd: password of user connecting to Usher controller @type passwd: string return: Deferred which returns and L{UsherResult} rtype: twisted.internet.defer.Deferred """ # serialize access to connect if not self.connect_lock.acquire(False): raise UsherEx, "Connection Pending" if self.factory: self.disconnect() # reset vms self.vms = {} # reset passwd to None until we're connected self.passwd = None self.factory = APIFactory() self.factory.api = self # connect to server reactor.connectSSL(ctrl_host, ctrl_port, self.factory, ssl.ClientContextFactory()) d = self.factory.login(credentials.UsernamePassword(user, passwd), client=self) d.addErrback(self._handle_err, misc.whoami()) d.addCallback(self._connected_to_ctrl, ctrl_host, ctrl_port, user, passwd) return d # start def start(self, cluster=None, prefix='', vmlist=None, count=None, first=None, last=None, dregex=None, dlist=None, dryrun=False, eid=None, **kw): """Start a VM or set of VMs @param cluster: Name of cluster to which VMs should belong @type cluster: string @param prefix: prefix to prepend to VM names @type prefix: string @param vmlist: list of strings of VMs to start @type vmlist: list of strings @param count: Number of consecutive VMs to start @type count: int @param first: Lowest number of VM to start @type first: int @param last: Highest number of VM to start @type last: int @param dregex: regular expression to match against destination LNM names @type dregex: list of strings @param dlist: list of strings to match against destination LNM names @type dlist: list of strings @param dryrun: Show what would have happened if the command had been run without the dryrun command (at the instant that the dryrun command was run of course) @type dryrun: bool @param eid: mechanism for admin user to be able to specify an alternate username for forming suffix for regex @type eid: string @param kw: Extra arguments passed to controller for use by plugins or LNM. Any keywords starting with 'vm_' are added to VM's uargs parameter. Note, the controller actually checks for two kw arguments: ram and ip_addrs. Though these are, strictly speaking, VM parameters, they're considered important enough that the controller checks for them. @type kw: dict @return: Deferred which returns an L{UsherResult} @rtype: twisted.internet.defer.Deferred @raise UsherNotConnected: Raised when not connected to Usher Controller @raise UsherPBDeadRefEx: Raised when capability for Usher Controller goes stale. Must reconnect when this happens @raise UsherInvalidInputEx: Raised when an invalid input or combination of parameters is received. @attention: The following conditions must hold: 1. first, last, and count must be non-negative 2. specifying first requires last or count 3. cannot use count with vmlist 4. cannot use count with both first and last """ # make sure we're connected if not self.aref: raise UsherNotConnectedEx # combine these to reduce params if ((first and first < 0) or (last and last < 0) or (count and count < 0)): raise UsherInvalidInputEx( "first, last, and count must be non-negative") vmrange = (first,last,count) # arg check self._arg_check(vmlist, vmrange) # get final destination list dlist = self._namecomplete(dlist, suffix=self.suffix) self._check_lnmlist(dlist) dregex = self._regex_append(dregex, dlist) dlist = self._regex_filter(dregex, self.lnms.keys()) # complete names (if necessary) in the vmlist vmlist = self._namecomplete(vmlist, cluster, eid) # check that the vms specified don't already exist self._check_vmlist(vmlist, isin=True) vmlist = self._get_vm_names_to_start(vmlist, cluster, prefix, vmrange, eid) # controller handles eid from kw, so add it if eid: kw['eid'] = eid if dryrun: d = UsherResult('dryrun', msg = 'started:' + linesep + linesep.join(vmlist)) return d try: if vmlist: d = self.aref.callRemote("start", vmlist, dlist, **kw) d.addErrback(self._handle_err, misc.whoami()) else: raise UsherInvalidInputEx( "No VMs started. No unique VMs specified in command") return d except pb.DeadReferenceError, ex: raise UsherPBDeadRefEx If you want, all the package is at http://usher.ucsd.edu/downloads/snapshots/usher/latest/usher-latest.tgz in the client folder you'll find the api.py Thank you very much hoping someone will help this totally inexpert user, Luigi. [View Less]

2 7

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com May 17, 2009

May 17, 2009

1 0

[Twisted-Python] How to make a service dependent of another?
by coder_gus May 15, 2009

May 15, 2009

Hi, I have a pb client and a tcp server. How can I make the server start only after/if the client started? Thanks.

3 2

Re: [Twisted-Python] How to make a service dependent of another?
by Terry Jones May 15, 2009

May 15, 2009

>>>>> "Gus" == coder gus <coder_gus(a)lavabit.com> writes: Gus> I have a pb client and a tcp server. How can I make the server start Gus> only after/if the client started? Hi Gus You might take a look at the code here http://www.twistedmatrix.com/pipermail/twisted-python/2009-February/019249.… and if you go back a couple of mails in that thread you'll see some more discussion. BTW, this approach hasn't received much comment, so you shouldn't assume it's … [View More]

1 0

Re: [Twisted-Python] SPAM-LOW: Re: Conch Checkers Help
by asset May 15, 2009

May 15, 2009

I tried that checker, but it failed authentication. Im not sure if theres a way to tell it which users can authenticate or if it just succeeds with any valid user, but it would'nt go through. I'm also not sure if it was intended for linux distros like Ubuntu (which is what im testing on). I even tried running my application via sudo in case the OS was blocking auth tests from a non priviledged user. I think ill do some manual tests with the pwd module it uses in order to drill out why its … [View More]

1 0

[Twisted-Python] Conch Checkers Help
by asset May 15, 2009

May 15, 2009

Can somebody point me into the right direction in how to switch out checkers.InMemoryUsernamePasswordDatabaseDontUse with something that would authenticate linux users, like root? On some things i just cant seem to get past documentation and api docs. Hoping for a slight push in the right direction. Also doesnt help that im new to linux. Any help would be greatly appreciated. BTW, Im going to be working on a multiline version of conch.recvline to handle edit chars when transitioning … [View More]

2 1

Re: [Twisted-Python] Consistent interfaces to asynchronous partially-available services using Deferreds and state machines (was Re: Another approach to allowing __init__ to work with Deferreds)
by Terry Jones May 14, 2009

May 14, 2009

>>>>> "Phil" == Phil Christensen <phil(a)bubblehouse.org> writes: Phil> I don't know if I agree with the need for such a feature (that is, Phil> deferred __init__ usage), but it was a very interesting coding Phil> challenge I wanted to take a whack at. I *think* I might have found a Phil> solution, but I don't know if it falls under the heading of Phil> "decorator abuse" ;-) Hi Phil I finally had time to look at your solution a bit (though I've not run it).… [View More]

2 1

[Twisted-Python] XMPP/Jabber server example
by Garret Heaton May 14, 2009

May 14, 2009

I'm trying to create an XMPP server using twisted.words and am having difficulty finding the correct way to make use of the classes provided. I was hoping someone might have an example or short explanation of which classes would be used to create a very basic server (using the latest classes<http://twistedmatrix.com/documents/8.2.0/api/twisted.words.protocols.jabber…>from 8.2.0). So far I have a t.w.p.j.xmlstream.XmlStreamServerFactory object creating t.w.x.xmlstream.XmlStream protocol … [View More]

3 3

[Twisted-Python] How do I turn off certain log messages?
by Reza Lotun May 14, 2009

May 14, 2009

Hi, I was just wondering about the best way to customize twisted logging - specifically, I'd like to remove the: "Starting factory.." and "Stopping factory.." log messages that pop up whenever I use twisted.web.client HTTPClientFactory. Is it simply a matter of writing a custom Logging Observer and specifically filtering those kinds of messages? I run a twisted.web server with twistd (with DailyLogFile) that for every connection to it makes a number of web-service calls using web.client. I … [View More]

3 4

[Twisted-Python] Recompiling _epoll.pyx
by Reza Lotun May 14, 2009

May 14, 2009

Hi, I noticed in the documentation for _epoll.pyx that the corresponding .c file was generated using pyrex 0.9.4.1. On a whim I downloaded 0.9.8.5 and regenerated the file and rebuilt python. I'm currently testing my server but noticing nothing out of the ordinary. Before I march on my merry way I was wondering if there could be any potential pitfall of using the non-standard _epoll.c? I only though to try it because - I'm using Python 2.6 which I compiled from source - I'm using Twisted … [View More]

1 1

Re: [Twisted-Python] can't get a simple client/server example to work!
by bbxx789_05ss＠yahoo.com May 14, 2009

May 14, 2009

Hi, --- On Wed, 5/13/09, Adam <adam.bcn(a)gmail.com> wrote: > From: Adam <adam.bcn(a)gmail.com> > Subject: Re: [Twisted-Python] can't get a simple client/server example to work! > To: "Twisted general discussion" <twisted-python(a)twistedmatrix.com> > Date: Wednesday, May 13, 2009, 1:08 AM > 2009/5/13 bbxx789_05ss(a)yahoo.com > <bbxx789_05ss(a)yahoo.com> > > > Here is the server > code:-------------from > twisted.internet.protocol … [View More]

2 1

Re: [Twisted-Python] can't get a simple client/server example to work!
by bbxx789_05ss＠yahoo.com May 13, 2009

May 13, 2009

Hi, --- On Wed, 5/13/09, Itamar Shtull-Trauring <itamar(a)itamarst.org> wrote: > > Have you read the client-side howto? > > http://twistedmatrix.com/projects/core/documentation/howto/clients.html > > Have you read my email?

1 0

[Twisted-Python] can't get a simple client/server example to work!
by bbxx789_05ss＠yahoo.com May 13, 2009

May 13, 2009

Here is the server code:-------------from twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactorimport sys class MyServer(Protocol): def connectionMade(self): self.transport.write("Hello") factory = Factory()factory.protocol = MyServer reactor.listenTCP(2000, factory) reactor.run()----------- Here is the client code:--------------from twisted.internet.protocol import Protocol, ClientCreatorfrom twisted.… [View More]

3 2

[Twisted-Python] installation error
by Alec Matusis May 13, 2009

May 13, 2009

Installing 8.2.0: creating /web20/encap/python-2.4.4/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-li nux-x86_64.egg Extracting Twisted-8.2.0-py2.4-linux-x86_64.egg to /web20/encap/python-2.4.4/lib/python2.4/site-packages File "/web20/encap/python-2.4.4/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-l inux-x86_64.egg/twisted/test/generator_failure_tests.py", line 66 yield ^ SyntaxError: invalid syntax Adding Twisted 8.2.0 to easy-install.pth file I think … [View More]

2 2

[Twisted-Python] Another approach to allowing __init__ to work with Deferreds
by Terry Jones May 12, 2009

May 12, 2009

I posted to this list back in Nov 2008 with subject: A Python metaclass for Twisted allowing __init__ to return a Deferred Briefly, I was trying to find a nice way to allow the __init__ method of a class to work with deferreds in such a way that methods of the class could use work done by __init__ safe in the knowledge that the deferreds had completed. E.g., if you have class X(object): def __init__(self, host, port): def final(connection): self.db = … [View More]connection d = makeDBConnection(host, port) d.addCallback(final) def query(self, q): return self.db.runQuery(q) Then when you make an X and call query on it, there's a chance the deferred wont have fired, and you'll get an error. This is just a very simple illustrative example. There are many more, and this is a general problem of the synchronous world (in which __init__ is supposed to prepare a fully-fledged class instance and cannot return a deferred) meeting the asynchronous world in which we would like to (and must) use deferreds. The earlier thread: http://twistedmatrix.com/pipermail/twisted-python/2008-November/018600.html Although I learned a lot in that thread, I wasn't completely happy with any of the solutions. Some of the things that still bugged me are in posts towards the end of the thread: http://twistedmatrix.com/pipermail/twisted-python/2008-November/018624.html http://twistedmatrix.com/pipermail/twisted-python/2008-November/018634.html The various approaches we took back then all boiled down to waiting for a deferred to fire before the class instance was fully ready to use. When that happened, you had your instance and could call its methods. I had also thought about an alternate approach: having __init__ add a callback to the deferreds it dealt with to set a flag in self and then have all dependent methods check that flag to see if the class instance was ready for use. But that 1) is ugly (too much extra code); 2) means the caller has to be prepared to deal with errors due to the class instance not being ready, and 3) adds a check to every method call. It would look something like this: class X(object): def __init__(self, host, port): self.ready = False def final(connection): self.db = connection self.ready = True d = makeDBConnection(host, port) d.addCallback(final) def query(self, q): if not self.ready: raise IAmNotReadyException() return self.db.runQuery(q) That was too ugly for my taste, for all of the above reasons, most especially for forcing the unfortunate caller of my code to handle IAmNotReadyException. Anyway.... fast forward 6 months and I've hit the same problem again. It's with existing code, in which I would like an __init__ to call something that (now, due to changes elsewhere) returns a deferred. So I started thinking again, and came up with a much cleaner way to do the alternate approach via a class mixin: from twisted.internet import defer class deferredInitMixin(object): def wrap(self, d, *wrappedMethods): self.waiting = [] self.stored = {} def restore(_): for method in self.stored: setattr(self, method, self.stored[method]) for d in self.waiting: d.callback(None) def makeWrapper(method): def wrapper(*args, **kw): d = defer.Deferred() d.addCallback(lambda _: self.stored[method](*args, **kw)) self.waiting.append(d) return d return wrapper for method in wrappedMethods: self.stored[method] = getattr(self, method) setattr(self, method, makeWrapper(method)) d.addCallback(restore) You use it as in the class Test below: from twisted.internet import defer, reactor def fire(d, value): print "I finally fired, with value", value d.callback(value) def late(value): d = defer.Deferred() reactor.callLater(1, fire, d, value) return d def called(result, what): print 'final callback of %s, result = %s' % (what, result) def stop(_): reactor.stop() class Test(deferredInitMixin): def __init__(self): d = late('Test') deferredInitMixin.wrap(self, d, 'f1', 'f2') def f1(self, arg): print "f1 called with", arg return late(arg) def f2(self, arg): print "f2 called with", arg return late(arg) if __name__ == '__main__': t = Test() d1 = t.f1(44) d1.addCallback(called, 'f1') d2 = t.f1(33) d2.addCallback(called, 'f1') d3 = t.f2(11) d3.addCallback(called, 'f2') d = defer.DeferredList([d1, d2, d3]) d.addBoth(stop) reactor.run() Effectively, the __init__ of my Test class asks deferredInitMixin to wrap some of its methods. deferredInitMixin stores the original methods away and replaces each of them with a function that immediately returns a deferred. So after __init__ finishes, code that calls the now-wrapped methods of the class instance before the deferred has fired will get a deferred back as usual (but see * below). As far as they know, everything is normal. Behind the scenes, deferredInitMixin has arranged for these deferreds to fire only after the deferred passed from __init__ has fired. Once that happens, deferredInitMixin also restores the original functions to the instance. As a result there is no overhead later to check a flag to see if the instance is ready to use. If the deferred from __init__ happens to fire before any of the instance's methods are called, it will simply restore the original methods. Finally (obviously?) you only pass the method names to deferredInitMixin that depend on the deferred in __init__ being done. BTW, calling the methods passed to deferredInitMixin "wrapped" isn't really accurate. They're just temporarily replaced. I quite like this approach. It's a second example of something I did in http://twistedmatrix.com/pipermail/twisted-python/2009-April/019522.html in which a pool of deferreds is accumulated and they're all fired when another deferred fires. It's nice because you don't reply with an error and there's no need for locking or other form of coordination - the work you need done is already in progress, so you get back a fresh deferred and everything goes swimmingly. * Minor note: the methods you wrap should probably be ones that already return deferreds. That way you always get a deferred back from them, whether they're temporarily wrapped or not. The above mixin works just fine if you ask it to wrap non-deferred-returning functions, but you have to deal with the possibility that they will return a deferred (i.e., if you call them while they're wrapped). Comments welcome / wanted. Terry [View Less]

3 2

Re: [Twisted-Python] Another approach to allowing __init__ to work with Deferreds
by Terry Jones May 12, 2009

May 12, 2009

Hi again Drew I realized I could have added another reason why I like my approach (for my current situation): I'm changing an existing (i.e., deployed) class so that its __init__ calls something returning a deferred which some instance methods of my class rely on having fired. I already have other code that uses my class, and that code is all written to use my class in the regular way. I could go change all those calls, but that gets me into a further mess as some of those calls are in the … [View More]

1 0