glyph@divmod.com wrote:

...

...

Is there going to be any facility, in newpb, for splitting up (de)serialization in small chunks?

As warner said, "yes".

Woohoo!

...

Yes, PB (de)serialization seems to be where I get responsiveness burps too. I can split up large processing chunks of my own (probably along the line of JP's Cooperator), but I don't see how to readily do that to delays that are internal to Twisted, without maintaining my own version or shunting all networking into a seperate thread (which I'm loath to do).

"shunting all networking into a separate thread" will certainly not work, since all of the APIs that jelly calls are not thread-safe. So you can rest easy in the knowledge that what you want to do is not only disgusting, it's impossible :).

However, the problem here is that certain code in Twisted is not as awesome as it possibly could be. While I agree this is sad, there isn't really a solution, except "make it more awesome", which takes time, money, and effort, which nobody is contributing beyond an individual scale.

PB has been effectively unmaintained for almost 2 years now. It is showing its age. NewPB will fix several issues, but until someone wants to give Mr. Warner a staff and a budget for improving it, it's likely that it will have some ugly spots too.

In other words, you don't have to maintain your own version. You just have to maintain the one in Twisted ;-).

I had thought about this, but I'm pretty strapped for time as my startup is self funded. Plus, it's not exactly like I can just walk into your SVN repository and start hacking. ;-) On the other hand, for all the free mileage I've gotten out of Twisted I do owe you guys a certain debt! Perhaps I could help out on NewPB, especially since I could really use the Slicer code Brian was talking about. However, I'm in a bit of an awkward spot to do so right now though, as I'm not yet using Twisted 2.0 and my project isn't at a good point to upgrade libaries. Still, I've got to upgrade sometime... Is there a good place to read more about NewPB? Or is it just a matter of searching the maillist archives and SVN repository? -Jasper

glyph@divmod.com wrote:

...

...

Is there going to be any facility, in newpb, for splitting up (de)serialization in small chunks?

As warner said, "yes".

Woohoo!

...

Yes, PB (de)serialization seems to be where I get responsiveness burps too. I can split up large processing chunks of my own (probably along the line of JP's Cooperator), but I don't see how to readily do that to delays that are internal to Twisted, without maintaining my own version or shunting all networking into a seperate thread (which I'm loath to do).

"shunting all networking into a separate thread" will certainly not work, since all of the APIs that jelly calls are not thread-safe. So you can rest easy in the knowledge that what you want to do is not only disgusting, it's impossible :).

However, the problem here is that certain code in Twisted is not as awesome as it possibly could be. While I agree this is sad, there isn't really a solution, except "make it more awesome", which takes time, money, and effort, which nobody is contributing beyond an individual scale.

PB has been effectively unmaintained for almost 2 years now. It is showing its age. NewPB will fix several issues, but until someone wants to give Mr. Warner a staff and a budget for improving it, it's likely that it will have some ugly spots too.

In other words, you don't have to maintain your own version. You just have to maintain the one in Twisted ;-).

I had thought about this, but I'm pretty strapped for time as my startup is self funded. Plus, it's not exactly like I can just walk into your SVN repository and start hacking. ;-) On the other hand, for all the free mileage I've gotten out of Twisted I do owe you guys a certain debt! Perhaps I could help out on NewPB, especially since I could really use the Slicer code Brian was talking about. However, I'm in a bit of an awkward spot to do so right now though, as I'm not yet using Twisted 2.0 and my project isn't at a good point to upgrade libaries. Still, I've got to upgrade sometime... Is there a good place to read more about NewPB? Or is it just a matter of searching the maillist archives and SVN repository? -Jasper