Mailman 3 [Twisted-Python] Deferred in C++ - Twisted

Oct. 18, 2008

      Hi,

I've implemented Deferred in C++.  There is room for improvement and
optimization, but the basic core works.  My focus has been on trying
to make the best possible API experience for users.  The internal
implementation is confusing and full of complexity unrelated to the
problems being solved.  For example, because the implementation
relies on templates so heavily, order-of-definition is an issue.
Luckily most of the complexity stays hidden away leaving a mostly
easy-to-use API, even in tricky cases such as callback pipelines
that change the result type being passed from one callback to
another by the Deferred:

  int convert_string(std::string result) {
      std::cout << "Converting string " << result << std::endl;
      return ion::Convert::to_integer<int>(result);
  }

  int add_ten(int result) {
      std::cout << "Adding 10 to " << result
                << " and returning " << result + 10 << std::endl;
      return result + 10;
  }

  Deferred deferred = Deferred::create_deferred<std::string>();
  deferred.add_callback(&convert_string);
  deferred.add_callback(&add_ten);
  std::cout << "Calling succeed" << std::endl;
  deferred.succeed(std::string("13"));
  std::cout << "Callbacks finished" << std::endl;

Running this code will produce the following output:

  Calling succeed
  Converting string 13
  Adding ten to 13 and returning 23
  Callbacks finished

It's possible to use method pointers as callbacks, too:

  class CallbackHandler {
  public:
      int handle_result(int result) {
          std::cout << "Handling " << result << std::endl;
          return result;
      }
  };

  CallbackHandler handler;
  deferred.add_callback(&handler, &CallbackHandler::handle_result);

One of the constraints that the API mostly hides is that the return
and parameter types of every callback in the pipeline need to be
known at compile time.  add_callbacks, add_callback and add_errback
are template methods that provide implementations for different
combinations of function and method pointers.  They infer return and
parameter types, based on the inputs passed in, and use them to
create internal structures.

Inferring types breaks down when adding a callback that returns a
Deferred because there's no way to infer the result type of a
Deferred that may not exist yet.  In such cases, the result type of
the Deferred must be explicitly stated:

  class DeferredCallback {
  public:
      DeferredCallback()
          : deferred(Deferred::create_deferred<int>) {}

      Deferred get_deferred(int) {
          std::cout << "Returning deferred" << std::endl;
          return this->deferred;
      }

      Deferred deferred;
  };

  Deferred deferred = Deferred::create_deferred<std::string>();
  DeferredCallback callback;
  // Notice the explicit result type here:
  deferred.add_callback<int>(&callback, DeferredCallback::get_deferred);
  deferred.add_callback(&add_ten);
  std::cout << "Calling succeed" << std::endl;
  deferred.succeed(std::string("13"));
  std::cerr << "Calling deferred" << std::endl;
  callback.deferred.succeed(6)
  std::cout << "Callbacks finished" << std::endl;

Running this code will produce the following output:

  Calling succeed
  Returning deferred
  Calling deferred
  Adding ten to 6 and returning 16
  Callbacks finished

When a callback returns a Deferred, processing of the pipeline
pauses until a result comes available, which is why add_ten is only
called after the inner Deferred succeeds.  If you forget the
explicit result type when adding a Deferred returning callback
you'll get a compile-time error:

../../ion/defer.h: In member function 'void
ion::Deferred::add_callbacks(Instance*, ion::Deferred
(Instance::*)(Param), ion::Deferred
(Instance::*)(ion::trule<ion::Failure, ion::DeallocObject>)) [with
Instance = ion::tests::DeferredMethodHandler, Param = int32_t]':
test-defer.cpp:416:   instantiated from here
../../ion/defer.h:773: error: no matching function for call to
'ion::CompileTimeAssertion<false>::CompileTimeAssertion(ion::Deferred::add_callbacks(Instance*,
ion::Deferred (Instance::*)(Param), ion::Deferred
(Instance::*)(ion::trule<ion::Failure, ion::DeallocObject>)) [with
Instance = ion::tests::DeferredMethodHandler, Param =
int32_t]::STATIC_ASSERT_Deferred_result_type_must_be_specified&)'
../../ion/debug.h:164: note: candidates are:
ion::CompileTimeAssertion<false>::CompileTimeAssertion()
../../ion/debug.h:164: note:
ion::CompileTimeAssertion<false>::CompileTimeAssertion(const
ion::CompileTimeAssertion<false>&)
../../ion/defer.h: At global scope:
../../ion/defer.h:772: warning: unused parameter 'instance'
../../ion/defer.h:772: warning: unused parameter 'method_callback'
../../ion/defer.h:772: warning: unused parameter 'method_errback'

If you look *really* hard you'll see some helpful information
starting with STATIC_ASSERT.  Yes, that's a feature.

I've implemented a Failure object, but failure handling over all is
not very good.  Exceptions are tricky because, unless you're
expecting a particular kind of exception, their types become opaque
as soon as they're thrown.  There's room for improvement but there's
no obvious path to a good solution yet.

The behaviour of errbacks is slightly different than in the Python
implementation.  Errbacks receive a trule<Failure> (a scoped pointer
that the errback owns) and return a result suitable for the next
callback.  They can either (a) recover from the error and return a
result, (b) raise an exception to indicate a new error or (c) raise
an UnhandledFailureError exception with the failure object.
Callback processing continues after (a), while errback processing
continues after (b) or (c).  A new Failure object is created when
(b) occurs.

  int recover(trule<Failure>) {
      std::cout << "Recovering from failure" << std::endl;
      return 43;
  }

  Deferred deferred = Deferred::create_deferred<int>();

  deferred.add_errback(&recover);
  deferred.add_callback(&add_ten);
  std::cout << "Calling fail" << std::endl;
  deferred.fail(Exception("Error message!"));
  std::cout << "Callbacks finished" << std::endl;

Running this code will produce the following output:

  Calling fail
  Recovering from failure
  Adding ten to 43 and returning 53
  Callbacks finished

I'm not sure how practical this component would be in a non-trivial
application.  The reliance on templates would negatively affect
compile time, for one thing.  More importantly, I've found
deciphering the error messages produced by simple mistakes can be
tricky.  Nonetheless, my main motivation was to see if it was
possible at all and so I'm pleased that it works in the end.

In the future, I'd like to see what the performance difference is
between this version and the Python version, but I haven't
investigated that at all.  There are also memory-management related
features that could be added to ease using this in C++ that would be
interesting to explore.  Porting to Windows is another item on the
list of things that I'd like to do.

The code is part of a project called ion (I/O and Networking
library), which I use for experiments like this.  I haven't merged
the deferred branch yet, but will soon.  Until it's merged, it's on
Launchpad:

https://code.edge.launchpad.net/~jkakar/ion/deferred

It will most likely only build on Ubuntu Intrepid right now.

Thanks,
J.

[Twisted-Python] Deferred in C++

Jamu Kakar

Amaury Forgeot d'Arc

Matthew Glubb

Amaury Forgeot d'Arc

Matthew Glubb

Jamu Kakar

Amaury Forgeot d'Arc

Matthew Glubb

Amaury Forgeot d'Arc

Matthew Glubb

Jamu Kakar

tags

participants (3)