A Twisted Design Decision

[koranthala]

On Jan 27, 9:27 pm, koranthala <koranth... at gmail.com> wrote:
> On Jan 27, 6:57 pm, Jean-Paul Calderone <exar... at divmod.com> wrote:
>
>
>
> > On Tue, 27 Jan 2009 05:46:25 -0800 (PST), koranthala <koranth... at gmail.com> wrote:
> > >Twisted, being twisted in its behavior is causing quite a lot of
> > >confusion in design decisions.
>
> > I'm not sure I agree with your premise. ;)
>
> > >I will put forward a comparison of reactor and non-reactor patterns.
> > >The code is not exact - whatever is shown is the gist of it.
>
> > >For example, a message handler - in a usual scenario:
> > >class messageHandler:
> > >   def run():
> > >         msg = self.get_next_msg()
> > >         if not msg.send():
> > >             self.handle_failure()
>
> > >To handle parallel execution, we will have to use threads, but the
> > >code flow is similar.
>
> > >How do we do the same in a reactor pattern (Twisted)?
> > >msg.send will cause a deferred to be raised - the failure, if it
> > >happens will happen much later.
> > >i.e. other than sending messageHandler object in msg.send(), I cannot
> > >see any mechanism of running handle_failure.
>
> > >In Twisted:
> > >class messageHandler:
> > >   def run():
> > >         msg = self.get_next_msg()
> > >         msg.send(self):
>
> > >class msgClass:
> > >    def send(o):
> > >        d = deferred.addCallBack(success_handler, o).addErrBack
> > >(failure_handler, o)
>
> > >    def failure_handler(o):
> > >        o.handle_failure()
>
> > This doesn't look like a correct or faithful translation of the original.
> > Here are two possibilities.  First:
>
> >     class messageHandler:
> >         def run():
> >             msg = self.get_next_msg()
> >             d = msg.send()
> >             def cbSendFailed(result):
> >                 if not result:
> >                     self.handle_failure()
> >             d.addErrback(cbSendFailed)
> >             return d
>
> > Next:
>
> >     class messageHandler:
> >         @inlineCallbacks
> >         def run():
> >             msg = self.get_next_msg()
> >             if not (yield msg.send()):
> >                 self.handle_failure()
>
> > These are both just straight translations from your version so as to
> > be able to handle a Deferred from `msg.send´.
>
> > >Basically, what I find is that a lot of functional encapsulation is
> > >now lost by following reactor pattern. handle_failure is
> > >messageHandlers code and makes for pretty viewing if called from
> > >inside messageHandler itself. But, due to the twisted nature of
> > >reactor pattern, the msg Class - who is functionally a lower class to
> > >messageHandler invoking messageHandler's code.
>
> > You don't need to lose anything.  I don't know what your motivation was
> > for re-arranging the code when you wrote the Twisted version, but it doesn't
> > appear to have been necessary.
>
> > >Is there a way to solve this in a more beautiful way? Am I missing
> > >something here?
>
> > Hope this helps,
>
> > Jean-Paul
>
> Thank you Jean-Paul.
> My code is more complex than what I have mentioned. When I mentioned
> msg.send, the msg object actually gets the data from DB etc to send.
> And there are many other items being done.
> I will try to see whether I can change the code to incorporate what
> you mentioned.
>
> I rewrote most of my code after learning just raw deferreds - I had
> planned to study inlineCallbacks - but then it slipped my mind  - now
> it has come to bit me. :-(

Hi,
  I tried to update the code as per the suggestion, but to no avail.
  My system uses Python2.4.3 (cannot move to 2.5) - so I tried to
rewrite with deferredGenerators - since I thought inlineCallbacks are
similar to deferredGenerators.

  But I cannot seem to rewrite it in a format where the functional
encapsulation is not broken.
  i.e. as I mentioned in the first example - I have to pass SELF to
child objects for them to modify it.

  The code was not exactly as I mentioned. I will try to explain more
below:
  The code before Twisted was incorporated.

  class MessageHandler:
    def send_message():
      if self.execute():  #Lots of checks going inside this
        for i in self.msgs: #Sends many messages at the same time
           if msg.send():
             self.success += 1
           else
             self.failure += 1

  class Message:
    def send():
      self.update_data() #The data to be sent is updated here
      return self.protocol.send() #Any protocol - for this example
HTTP is used

  class Protocol:
     def send():
        HTTP get page
        if page received:
          parse page and see parameters
          if parameters:
            return True
        return False

The code I rewrote after Twisted was incorporated:

  class MessageHandler:
    def send_message():
      if self.execute():  #Lots of checks going inside this
        for i in self.msgs: #Sends many messages at the same time
           msg.send(self):  #Has to send myself to the childclass

  class Message:
    def send(h):
      self.h = h   #The message handler object
      self.update_data() #The data to be sent is updated here
      return self.protocol.send(self) #Again, sending myself to child
class

    def update(status):
      if status:
        self.h.success += 1
      else:
        self.h.failure += 1

  class Protocol:
     @deferredGenerator
     def send(msg):
        d = waitForDeferred(getPage(url, method, data))
        yield d
        if page received:
          parse page and see parameters
          if parameters:
            msg.update(True)
        msg.update(False)

   As I mentioned, I am unable to rewrite it in reactor pattern
without breaking functional encapsulation.
   I tried re-writing it as Jean-Paul mentioned - trying to push the
deferred to msg.send etc, but then functional encapsulation of
Protocol was broken.
   Am I being a blockhead here or is it possible to rewrite it in a
proper mechanism?