PB: remote_* deferred return magic
![](https://secure.gravatar.com/avatar/9e1312e82732ddc84c53cabd2ef975b4.jpg?s=120&d=mm&r=g)
I'm playing with PB, and it's booth cool and somewhat confusing. So I have some remote_* methods, e.g. def remote_ping(self): return 'pong' def remote_run(self): d = Deferred() d.AddCallback(self.got_result) # return d # <- magic here when deferred is returned? def got_result(self, res) return res When I do callRemote('ping') and execute callback, I am getting 'pong' as a result. However, when I do callRemote('run'), it returns a deferred which returns None, which is expected, as remote_run() doesn't have any result yet. However, if I return deferred ("d") from remote_run, then callRemote('run') gives me the result. Is it some expected magic going on when remote_ method returns a deferred?
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Fri, Jan 10, 2025 at 10:39 AM Kirill Miazine <km@krot.org> wrote:
First, while PB probably is objectively pretty cool, you almost certainly shouldn't use it. The complexity isn't warranted except perhaps for a very few specific kinds of applications. Even for those applications, the level of development on PB means that many features you will likely want aren't available and even those features which are available have implementation shortcomings that aren't likely to be addressed. *IMO* the entire package should have been split out of Twisted long ago. As implied by the above, the right choice of protocol for your application does depend on your application's specific requirements. That said, HTTP is a pretty good fit for a wide range of common application types. Another protocol Twisted supports, AMP, covers some more ground - though you might find wider support for similar protocols such as those based on ProtoBufs or msgpack (AMP is a Twisted invention and though it has a number of very nice properties, you won't find many people using it). To answer the question you asked, there is a limited amount of magic going on that allows you to return a Deferred and have the method behave as though it returned the result that Deferred eventually fires with, instead. This is a common pattern when developing with Twisted as it makes dealing with asynchronous implementations more convenient. In fact, it's essentially the reason Deferred exists at all. You'll find this feature in Twisted's implementation of AMP and HTTP and many other protocols as well. Jean-Paul
![](https://secure.gravatar.com/avatar/9e1312e82732ddc84c53cabd2ef975b4.jpg?s=120&d=mm&r=g)
• Jean-Paul Calderone [2025-01-10 16:56]:
Ouch. Tanks for the warning. Are there other parts of Twisted which should be avoided for new stuff?
I came from HTTP. In fact, the PB remote_run receives some pickled data and HMAC digest, and the response is constructed in a similar way. So PB is just a way to avoid HTTP.
Thanks for the AMP pointer, it seems more modern. I'll give it a try.
Yes, it felt very logical, but I didn't find it mentioned in the docs, thus a need for a confirmation.
You'll find this feature in Twisted's implementation of AMP and HTTP and many other protocols as well.
I love it!
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I mean you probably don't want to base any big new things on POP3 :). PB and AMP are the two protocols that are Twisted's attempt at a "native", minimal, interprocess communication mechanism; all the other protocols are from existing specifications.
As implied by the above, the right choice of protocol for your application does depend on your application's specific requirements. That said, HTTP is a pretty good fit for a wide range of common application types.
I came from HTTP. In fact, the PB remote_run receives some pickled data and HMAC digest, and the response is constructed in a similar way. So PB is just a way to avoid HTTP.
If you like the way that PB feels to work with, there are things we could do to modernize it (which mostly would look like making a version of AMP that would work based off of type annotations) which I would be happy to work with you on. It might be fun to write up the problems with PB for their own sake.
Another protocol Twisted supports, AMP, covers some more ground - though you might find wider support for similar protocols such as those based on ProtoBufs or msgpack (AMP is a Twisted invention and though it has a number of very nice properties, you won't find many people using it).
Thanks for the AMP pointer, it seems more modern. I'll give it a try.
The transition is not clean, because there were a few PB users who were unlikely to migrate to AMP right away due to the weight of legacy (does anyone know if buildbot still uses PB?), but AMP was effectively the "next version" of PB.
To answer the question you asked, there is a limited amount of magic going on that allows you to return a Deferred and have the method behave as though it returned the result that Deferred eventually fires with, instead. This is a common pattern when developing with Twisted as it makes dealing with asynchronous implementations more convenient. In fact, it's essentially the reason Deferred exists at all.
Yes, it felt very logical, but I didn't find it mentioned in the docs, thus a need for a confirmation.
You'll find this feature in Twisted's implementation of AMP and HTTP and many other protocols as well.
I love it!
The HTTP server somewhat infamously does not do this by default, still leaning on the NOT_DONE_YET constant, but various places do, including all of Klein. This is just because of the big hairball of "we need a whole new HTTP resource model" <https://github.com/twisted/twisted/issues/4688> that has been a known issue for (ugghhhh) 20 years. Again, always happy to have a volunteer to help get some momentum on that :) -g
![](https://secure.gravatar.com/avatar/9e1312e82732ddc84c53cabd2ef975b4.jpg?s=120&d=mm&r=g)
• Glyph [2025-01-11 07:16]:
POP3 is so simple that it could be done with LineOnlyReceiver in an evening. I'm asking as I now see that PB shouldn't be used, and (think) I also read (somewhere) something about Words?
I did give AMP a try yesterday, and getting started with PB was a lot easier. That's to be expected, given that AMP is supposed to be a lower level protocol than PB (does it?), but I met again pre-3 ghosts: amp.String() is actually bytes, and amp.Unicode() is actually string. It caused further confusion that definition of commands require that key names are specified as b'' "strings", but when returning a dict from the commands, key names are ordinary '' strings. Furthermore, I am a bit concerned about the 65535 bytes size limit on the values. I could dissect the objects I am passing and reconstruct them upon receiving. I am willing to give it a try, or just stop bothering and keep using HTTP.
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Fri, Jan 10, 2025 at 10:39 AM Kirill Miazine <km@krot.org> wrote:
First, while PB probably is objectively pretty cool, you almost certainly shouldn't use it. The complexity isn't warranted except perhaps for a very few specific kinds of applications. Even for those applications, the level of development on PB means that many features you will likely want aren't available and even those features which are available have implementation shortcomings that aren't likely to be addressed. *IMO* the entire package should have been split out of Twisted long ago. As implied by the above, the right choice of protocol for your application does depend on your application's specific requirements. That said, HTTP is a pretty good fit for a wide range of common application types. Another protocol Twisted supports, AMP, covers some more ground - though you might find wider support for similar protocols such as those based on ProtoBufs or msgpack (AMP is a Twisted invention and though it has a number of very nice properties, you won't find many people using it). To answer the question you asked, there is a limited amount of magic going on that allows you to return a Deferred and have the method behave as though it returned the result that Deferred eventually fires with, instead. This is a common pattern when developing with Twisted as it makes dealing with asynchronous implementations more convenient. In fact, it's essentially the reason Deferred exists at all. You'll find this feature in Twisted's implementation of AMP and HTTP and many other protocols as well. Jean-Paul
![](https://secure.gravatar.com/avatar/9e1312e82732ddc84c53cabd2ef975b4.jpg?s=120&d=mm&r=g)
• Jean-Paul Calderone [2025-01-10 16:56]:
Ouch. Tanks for the warning. Are there other parts of Twisted which should be avoided for new stuff?
I came from HTTP. In fact, the PB remote_run receives some pickled data and HMAC digest, and the response is constructed in a similar way. So PB is just a way to avoid HTTP.
Thanks for the AMP pointer, it seems more modern. I'll give it a try.
Yes, it felt very logical, but I didn't find it mentioned in the docs, thus a need for a confirmation.
You'll find this feature in Twisted's implementation of AMP and HTTP and many other protocols as well.
I love it!
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I mean you probably don't want to base any big new things on POP3 :). PB and AMP are the two protocols that are Twisted's attempt at a "native", minimal, interprocess communication mechanism; all the other protocols are from existing specifications.
As implied by the above, the right choice of protocol for your application does depend on your application's specific requirements. That said, HTTP is a pretty good fit for a wide range of common application types.
I came from HTTP. In fact, the PB remote_run receives some pickled data and HMAC digest, and the response is constructed in a similar way. So PB is just a way to avoid HTTP.
If you like the way that PB feels to work with, there are things we could do to modernize it (which mostly would look like making a version of AMP that would work based off of type annotations) which I would be happy to work with you on. It might be fun to write up the problems with PB for their own sake.
Another protocol Twisted supports, AMP, covers some more ground - though you might find wider support for similar protocols such as those based on ProtoBufs or msgpack (AMP is a Twisted invention and though it has a number of very nice properties, you won't find many people using it).
Thanks for the AMP pointer, it seems more modern. I'll give it a try.
The transition is not clean, because there were a few PB users who were unlikely to migrate to AMP right away due to the weight of legacy (does anyone know if buildbot still uses PB?), but AMP was effectively the "next version" of PB.
To answer the question you asked, there is a limited amount of magic going on that allows you to return a Deferred and have the method behave as though it returned the result that Deferred eventually fires with, instead. This is a common pattern when developing with Twisted as it makes dealing with asynchronous implementations more convenient. In fact, it's essentially the reason Deferred exists at all.
Yes, it felt very logical, but I didn't find it mentioned in the docs, thus a need for a confirmation.
You'll find this feature in Twisted's implementation of AMP and HTTP and many other protocols as well.
I love it!
The HTTP server somewhat infamously does not do this by default, still leaning on the NOT_DONE_YET constant, but various places do, including all of Klein. This is just because of the big hairball of "we need a whole new HTTP resource model" <https://github.com/twisted/twisted/issues/4688> that has been a known issue for (ugghhhh) 20 years. Again, always happy to have a volunteer to help get some momentum on that :) -g
![](https://secure.gravatar.com/avatar/9e1312e82732ddc84c53cabd2ef975b4.jpg?s=120&d=mm&r=g)
• Glyph [2025-01-11 07:16]:
POP3 is so simple that it could be done with LineOnlyReceiver in an evening. I'm asking as I now see that PB shouldn't be used, and (think) I also read (somewhere) something about Words?
I did give AMP a try yesterday, and getting started with PB was a lot easier. That's to be expected, given that AMP is supposed to be a lower level protocol than PB (does it?), but I met again pre-3 ghosts: amp.String() is actually bytes, and amp.Unicode() is actually string. It caused further confusion that definition of commands require that key names are specified as b'' "strings", but when returning a dict from the commands, key names are ordinary '' strings. Furthermore, I am a bit concerned about the 65535 bytes size limit on the values. I could dissect the objects I am passing and reconstruct them upon receiving. I am willing to give it a try, or just stop bothering and keep using HTTP.
participants (4)
-
Glyph
-
Jean-Paul Calderone
-
Kirill Miazine
-
meejah