That's because SOAP and company suck. There is always, always, *always* a better way to accomplish a task than using SOAP. I've ranted about this topic before in another thread, to no avail, and I doubt I'll change your mind either, but someone needs to speak out against this nonsense.

Curious, what would you use if you needed a general remote programming interface built using python, where you can simply upload new python codelets, that should be accessible not just from python but from any major Windows and Unix PL, where you need complex types, and where you wouldn't know what kind of tasks you'll be solving with it in a couple of years. Corba? XML-RPC?

On Mon, 2006-06-19 at 10:11 -0400, Chaz. wrote:

Right now I support XML-RPC but people have clamored for SOAP, so SOAP it is as well.

It's really popular, especially among .NET disciples ... ;)

Personally I think sending stuff in ASCII text is silly, but it is out there today as (a more or less) standard.

Well, it's certainly not something that speeds up processing, but it makes systems much easier to debug, if the underlying messaging system doesn't work as it should. Which of course happens far more often with SOAP than with other systems. Actually, SOAP would be quite nice, if it 1) were standardised enough to really allow condition free interoperability, 2) were less bloated and 3) didn't simply ignore major parts of the actual RPC/Document request process in the standard. The WS-I people (http://www.ws-i.org/) try to address these things, in an attempt to "standardise the standard" and by adding additional stuff like WS-Security (standardised security credentials handling) and WS-Addressing (actually specifying the service endpoint/protocol inside a SOAP message). Maybe that will improve the situation somewhat, python's ZSI is at least tracking those developments. -Thomas

On Mon, 2006-06-19 at 10:43 -0400, washort@twistedmatrix.com wrote:

On Mon, Jun 19, 2006 at 01:19:58PM +0200, Thomas Jacob wrote:

...

Curious, what would you use if you needed a general remote programming interface built using python, where you can simply upload new python codelets, that should be accessible not just from python but from any major Windows and Unix PL, where you need complex types, and where you wouldn't know what kind of tasks you'll be solving with it in a couple of years.

Sure ;) But these codelets are supposed to offer some kind of RPC-Service, once they are uploaded, several different "rpc functions" per service, various complex input and output data types, in a multi platform/OS environment. And the question was: what is better suited for that than SOAP.

Yes, the ssh side worked well. I was hoping to give my customer something more graphical...lol. Peace, Chaz Jean-Paul Calderone wrote:

On Wed, 21 Jun 2006 09:28:07 -0400, "Chaz." <eprparadocs@gmail.com> wrote:

...

I've built a 'ManholeFactory' to use with my application (see below). When I try to connect to it via the 'manhole' client (under Linux) it asks for Host, port, Name, Password, all of which I provide. But it gets an error returned.

I was wondering if I need to do anything special (besides what my manhole support does)?

The manhole you set up with the attached code only listens for SSH connections. The Gtk2 manhole client that you're running hasn't been updated to run with the new version of manhole yet.

I've run your code and tried ssh'ing into it and everything seems to work alright.

Jean-Paul

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

I'm sure there will be a flood of answers for this (involving using a Producer/Consumer)...I'd give a clue, but my PB usage right now is only in quick messages... However, I wanted to ask if you could please create new messages for new threads on the list instead of replying to arbitrary old messages. When you start new threads by replying, it adds a In-Reply- To header that thwarts any attempts to sort by threads. This is my OCD talking, so please for give me... ;-) -phil On Jun 25, 2006, at 2:20 PM, Chaz. wrote:

I have a problem to solve: I need to get files from one machine to another. I had thought about all the obvious solutions (and implemented some of them); for instance adding an FTP server to my Twisted services and using a client. I thought about doing the file transfer with XML-RPC and even SOAP. And I even thought of WebDav.

I started reading about PB and thought it might be useful. I thought about building a "remote" class that simulates open and all its functions, like read, write, close, etc.

The more I thought about it the more I thought it cool and the way to do it. That got me thinking that I must be missing something. I am curious about what you might think of this approach. Is there another better way?

Peace, Chaz.

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

----- Original Message ----- From: "Chaz." <eprparadocs@gmail.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Sunday, June 25, 2006 2:20 PM Subject: [Twisted-Python] Need words of wisdom regarding PB

I have a problem to solve: I need to get files from one machine to another. I had thought about all the obvious solutions (and implemented some of them); for instance adding an FTP server to my Twisted services and using a client. I thought about doing the file transfer with XML-RPC and even SOAP. And I even thought of WebDav.

I started reading about PB and thought it might be useful. I thought about building a "remote" class that simulates open and all its functions, like read, write, close, etc.

The more I thought about it the more I thought it cool and the way to do it. That got me thinking that I must be missing something. I am curious about what you might think of this approach. Is there another better way?

i implemented file transfer with an optional intermediate gzip stage using pb a long while ago. if you're dealing with fairly low latency, a single pb connection is fine; for higher latencies, you'll want to multiplex multiple pb connections (this has to do with tcp, not pb). with my quick hack, i was able to saturate a fast-e with virtually no significant cpu utilization - the disk io and the network were the bottlenecks. -p

chaz, quick note: please don't top post (in replies to me, anyway). just common courtesy. in my context, the solution was meant as a synchronization method with some application-level logic to be used instead of rsync (i didn't need differencing and needed some complicated candidacy logic). the pb client simply opened multiple connections and sent data chunks labelled with chunk id and object id, multiplexed over the connection pool, with the chunks being pulled out of a queue that a threadpool reading from disk and optionally gzipping data was writing to. i didn't need to write logic to grow the connection pool to maximize bandwidth (i easily saturated a fast-e), but that would be trivial. note: i do not advocate doing this unless there is a very good reason existing tools won't work for your application. reinventing the wheel makes baby jesus cry. -p ----- Original Message ----- From: "Chaz." <eprparadocs@gmail.com> To: "Paul G" <paul-lists@perforge.com> Cc: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Wednesday, June 28, 2006 8:03 AM Subject: Re: [Twisted-Python] Need words of wisdom regarding PB

Paul,

Thanks for the information. I hadn't thought about multiple pb connections, but I might even for my low-latency environment. I can have files that range from very small to very very large and getting them transferred in a reasonable period of time might might require many connections at once.

In your solution did you just allow the client to "divide" up the work or was there something else you did?

Chaz

Paul G wrote:

...

----- Original Message ----- From: "Chaz." <eprparadocs@gmail.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Sunday, June 25, 2006 2:20 PM Subject: [Twisted-Python] Need words of wisdom regarding PB

...

I have a problem to solve: I need to get files from one machine to another. I had thought about all the obvious solutions (and implemented some of them); for instance adding an FTP server to my Twisted services and using a client. I thought about doing the file transfer with XML-RPC and even SOAP. And I even thought of WebDav.

I started reading about PB and thought it might be useful. I thought about building a "remote" class that simulates open and all its functions, like read, write, close, etc.

The more I thought about it the more I thought it cool and the way to do it. That got me thinking that I must be missing something. I am curious about what you might think of this approach. Is there another better way?

i implemented file transfer with an optional intermediate gzip stage using pb a long while ago. if you're dealing with fairly low latency, a single pb connection is fine; for higher latencies, you'll want to multiplex multiple pb connections (this has to do with tcp, not pb). with my quick hack, i was able to saturate a fast-e with virtually no significant cpu utilization - the disk io and the network were the bottlenecks.

-p

tzahi, it saddens me to say that you can't get it. the reason is simple: i can't get it either. i'd erroneously thought the code was backed up before i wiped a box, but that was not the case :( it is now lost, along with some other semi-useful stuff. with that said, i don't think it would've been useful for what you were attempting - i was compressing the data before sending it over pb, while you seem to be interested in doing this automagically at the pb or transport level. -p ----- Original Message ----- From: "Tzahi Fadida" <Tzahi.ML@gmail.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Wednesday, June 28, 2006 8:39 AM Subject: Re: [Twisted-Python] Need words of wisdom regarding PB On Wednesday 28 June 2006 14:50, Paul G wrote:

i implemented file transfer with an optional intermediate gzip stage using pb a long while ago. if you're dealing with fairly low latency, a single pb connection is fine; for higher latencies, you'll want to multiplex multiple pb connections (this has to do with tcp, not pb). with my quick hack, i was able to saturate a fast-e with virtually no significant cpu utilization - the disk io and the network were the bottlenecks.

I am also very much interested in compressing the underlying transfers using PB. Where can we get the hack? :) I am looking for something that is done after serialization so all features of jelly remains. I.e., just before transfer, compress the data. Preferably with some control, i.e., with each message i send i want to be able to say if to compress and to what level of compression. Currently i have to circumvent jelly by pickle and compress but i can only do that when server sends to client since the client trusts the server. However the server does not trust the client and thus, cannot allow pickled objects to be unserialized. maybe something like callRemoteCompressed... -- Regards, Tzahi. -- Tzahi Fadida Blog: http://tzahi.blogsite.org | Home Site: http://tzahi.webhop.info WARNING TO SPAMMERS: see at http://members.lycos.co.uk/my2nis/spamwarning.html _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

----- Original Message ----- From: <glyph@divmod.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Thursday, June 29, 2006 9:00 PM Subject: Re: [Twisted-Python] Need words of wisdom regarding PB

On Thu, 29 Jun 2006 20:18:14 -0400, Paul G <paul-lists@perforge.com> wrote:

...

i was compressing the data before sending it over pb, while you seem to be interested in doing this automagically at the pb or transport level.

This is, nevertheless, generally the correct way to compress data in PB. PB itself is so low-overhead that compression isn't going to buy you much in terms of overhead reduction, and jellying/unjellying something that is already a byte sequence is a waste of time. compress the data, transfer blocks of compressed data as PB strings - the strings are incredibly low overhead and length-prefixed so there isn't any real parsing overhead.

oh, i agree of course, not to mention the fact that compression (and disk reads, if you have any) has to be performed in a threadpool. i wasn't commenting on the validity of the OP's approach one way or another, since one usually needs to have thick skin to handle the resultant acrimony or a lot of free time to participate in the ensuing back-and-forth ;] -p

That's because SOAP and company suck. There is always, always, *always* a better way to accomplish a task than using SOAP. I've ranted about this topic before in another thread, to no avail, and I doubt I'll change your mind either, but someone needs to speak out against this nonsense.

Curious, what would you use if you needed a general remote programming interface built using python, where you can simply upload new python codelets, that should be accessible not just from python but from any major Windows and Unix PL, where you need complex types, and where you wouldn't know what kind of tasks you'll be solving with it in a couple of years. Corba? XML-RPC?

On Mon, 2006-06-19 at 10:11 -0400, Chaz. wrote:

Right now I support XML-RPC but people have clamored for SOAP, so SOAP it is as well.

It's really popular, especially among .NET disciples ... ;)

Personally I think sending stuff in ASCII text is silly, but it is out there today as (a more or less) standard.

Well, it's certainly not something that speeds up processing, but it makes systems much easier to debug, if the underlying messaging system doesn't work as it should. Which of course happens far more often with SOAP than with other systems. Actually, SOAP would be quite nice, if it 1) were standardised enough to really allow condition free interoperability, 2) were less bloated and 3) didn't simply ignore major parts of the actual RPC/Document request process in the standard. The WS-I people (http://www.ws-i.org/) try to address these things, in an attempt to "standardise the standard" and by adding additional stuff like WS-Security (standardised security credentials handling) and WS-Addressing (actually specifying the service endpoint/protocol inside a SOAP message). Maybe that will improve the situation somewhat, python's ZSI is at least tracking those developments. -Thomas

On Mon, 2006-06-19 at 10:43 -0400, washort@twistedmatrix.com wrote:

On Mon, Jun 19, 2006 at 01:19:58PM +0200, Thomas Jacob wrote:

...

Curious, what would you use if you needed a general remote programming interface built using python, where you can simply upload new python codelets, that should be accessible not just from python but from any major Windows and Unix PL, where you need complex types, and where you wouldn't know what kind of tasks you'll be solving with it in a couple of years.

Sure ;) But these codelets are supposed to offer some kind of RPC-Service, once they are uploaded, several different "rpc functions" per service, various complex input and output data types, in a multi platform/OS environment. And the question was: what is better suited for that than SOAP.

Yes, the ssh side worked well. I was hoping to give my customer something more graphical...lol. Peace, Chaz Jean-Paul Calderone wrote:

On Wed, 21 Jun 2006 09:28:07 -0400, "Chaz." <eprparadocs@gmail.com> wrote:

...

I've built a 'ManholeFactory' to use with my application (see below). When I try to connect to it via the 'manhole' client (under Linux) it asks for Host, port, Name, Password, all of which I provide. But it gets an error returned.

I was wondering if I need to do anything special (besides what my manhole support does)?

The manhole you set up with the attached code only listens for SSH connections. The Gtk2 manhole client that you're running hasn't been updated to run with the new version of manhole yet.

I've run your code and tried ssh'ing into it and everything seems to work alright.

Jean-Paul

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

I'm sure there will be a flood of answers for this (involving using a Producer/Consumer)...I'd give a clue, but my PB usage right now is only in quick messages... However, I wanted to ask if you could please create new messages for new threads on the list instead of replying to arbitrary old messages. When you start new threads by replying, it adds a In-Reply- To header that thwarts any attempts to sort by threads. This is my OCD talking, so please for give me... ;-) -phil On Jun 25, 2006, at 2:20 PM, Chaz. wrote:

I have a problem to solve: I need to get files from one machine to another. I had thought about all the obvious solutions (and implemented some of them); for instance adding an FTP server to my Twisted services and using a client. I thought about doing the file transfer with XML-RPC and even SOAP. And I even thought of WebDav.

I started reading about PB and thought it might be useful. I thought about building a "remote" class that simulates open and all its functions, like read, write, close, etc.

The more I thought about it the more I thought it cool and the way to do it. That got me thinking that I must be missing something. I am curious about what you might think of this approach. Is there another better way?

Peace, Chaz.

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

----- Original Message ----- From: "Chaz." <eprparadocs@gmail.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Sunday, June 25, 2006 2:20 PM Subject: [Twisted-Python] Need words of wisdom regarding PB

I have a problem to solve: I need to get files from one machine to another. I had thought about all the obvious solutions (and implemented some of them); for instance adding an FTP server to my Twisted services and using a client. I thought about doing the file transfer with XML-RPC and even SOAP. And I even thought of WebDav.

I started reading about PB and thought it might be useful. I thought about building a "remote" class that simulates open and all its functions, like read, write, close, etc.

The more I thought about it the more I thought it cool and the way to do it. That got me thinking that I must be missing something. I am curious about what you might think of this approach. Is there another better way?

i implemented file transfer with an optional intermediate gzip stage using pb a long while ago. if you're dealing with fairly low latency, a single pb connection is fine; for higher latencies, you'll want to multiplex multiple pb connections (this has to do with tcp, not pb). with my quick hack, i was able to saturate a fast-e with virtually no significant cpu utilization - the disk io and the network were the bottlenecks. -p

chaz, quick note: please don't top post (in replies to me, anyway). just common courtesy. in my context, the solution was meant as a synchronization method with some application-level logic to be used instead of rsync (i didn't need differencing and needed some complicated candidacy logic). the pb client simply opened multiple connections and sent data chunks labelled with chunk id and object id, multiplexed over the connection pool, with the chunks being pulled out of a queue that a threadpool reading from disk and optionally gzipping data was writing to. i didn't need to write logic to grow the connection pool to maximize bandwidth (i easily saturated a fast-e), but that would be trivial. note: i do not advocate doing this unless there is a very good reason existing tools won't work for your application. reinventing the wheel makes baby jesus cry. -p ----- Original Message ----- From: "Chaz." <eprparadocs@gmail.com> To: "Paul G" <paul-lists@perforge.com> Cc: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Wednesday, June 28, 2006 8:03 AM Subject: Re: [Twisted-Python] Need words of wisdom regarding PB

Paul,

Thanks for the information. I hadn't thought about multiple pb connections, but I might even for my low-latency environment. I can have files that range from very small to very very large and getting them transferred in a reasonable period of time might might require many connections at once.

In your solution did you just allow the client to "divide" up the work or was there something else you did?

Chaz

Paul G wrote:

...

----- Original Message ----- From: "Chaz." <eprparadocs@gmail.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Sunday, June 25, 2006 2:20 PM Subject: [Twisted-Python] Need words of wisdom regarding PB

...

I have a problem to solve: I need to get files from one machine to another. I had thought about all the obvious solutions (and implemented some of them); for instance adding an FTP server to my Twisted services and using a client. I thought about doing the file transfer with XML-RPC and even SOAP. And I even thought of WebDav.

I started reading about PB and thought it might be useful. I thought about building a "remote" class that simulates open and all its functions, like read, write, close, etc.

The more I thought about it the more I thought it cool and the way to do it. That got me thinking that I must be missing something. I am curious about what you might think of this approach. Is there another better way?

i implemented file transfer with an optional intermediate gzip stage using pb a long while ago. if you're dealing with fairly low latency, a single pb connection is fine; for higher latencies, you'll want to multiplex multiple pb connections (this has to do with tcp, not pb). with my quick hack, i was able to saturate a fast-e with virtually no significant cpu utilization - the disk io and the network were the bottlenecks.

-p

tzahi, it saddens me to say that you can't get it. the reason is simple: i can't get it either. i'd erroneously thought the code was backed up before i wiped a box, but that was not the case :( it is now lost, along with some other semi-useful stuff. with that said, i don't think it would've been useful for what you were attempting - i was compressing the data before sending it over pb, while you seem to be interested in doing this automagically at the pb or transport level. -p ----- Original Message ----- From: "Tzahi Fadida" <Tzahi.ML@gmail.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Wednesday, June 28, 2006 8:39 AM Subject: Re: [Twisted-Python] Need words of wisdom regarding PB On Wednesday 28 June 2006 14:50, Paul G wrote:

i implemented file transfer with an optional intermediate gzip stage using pb a long while ago. if you're dealing with fairly low latency, a single pb connection is fine; for higher latencies, you'll want to multiplex multiple pb connections (this has to do with tcp, not pb). with my quick hack, i was able to saturate a fast-e with virtually no significant cpu utilization - the disk io and the network were the bottlenecks.

I am also very much interested in compressing the underlying transfers using PB. Where can we get the hack? :) I am looking for something that is done after serialization so all features of jelly remains. I.e., just before transfer, compress the data. Preferably with some control, i.e., with each message i send i want to be able to say if to compress and to what level of compression. Currently i have to circumvent jelly by pickle and compress but i can only do that when server sends to client since the client trusts the server. However the server does not trust the client and thus, cannot allow pickled objects to be unserialized. maybe something like callRemoteCompressed... -- Regards, Tzahi. -- Tzahi Fadida Blog: http://tzahi.blogsite.org | Home Site: http://tzahi.webhop.info WARNING TO SPAMMERS: see at http://members.lycos.co.uk/my2nis/spamwarning.html _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

----- Original Message ----- From: <glyph@divmod.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Thursday, June 29, 2006 9:00 PM Subject: Re: [Twisted-Python] Need words of wisdom regarding PB

On Thu, 29 Jun 2006 20:18:14 -0400, Paul G <paul-lists@perforge.com> wrote:

...

i was compressing the data before sending it over pb, while you seem to be interested in doing this automagically at the pb or transport level.

This is, nevertheless, generally the correct way to compress data in PB. PB itself is so low-overhead that compression isn't going to buy you much in terms of overhead reduction, and jellying/unjellying something that is already a byte sequence is a waste of time. compress the data, transfer blocks of compressed data as PB strings - the strings are incredibly low overhead and length-prefixed so there isn't any real parsing overhead.

oh, i agree of course, not to mention the fact that compression (and disk reads, if you have any) has to be performed in a threadpool. i wasn't commenting on the validity of the OP's approach one way or another, since one usually needs to have thick skin to handle the resultant acrimony or a lot of free time to participate in the ensuing back-and-forth ;] -p