Whoops... forgot the attachments. Here they are. On Tue, 28 Dec 2004 14:32:04 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

I would really like to be able to pass custom exceptions over the wire in PB. Previous discussion on this list ended with the conclusion that it currently isn't possible without hacking a local copy of Twisted.

I have done that to get the result I want, with minimal testing, and would like to get some discussion going around this topic. Attached are 3 test files, s.py for the server, c.py for the client, and e.py for the error definitions which are imported by both s.py and c.py. If I make the following modification to my local copy of twisted/spread/pb.py in the CopyableFailure's getStateToCopy method, the custom error is passed back as I want.

#state['value'] = str(self.value) # Exception instance state['value'] = self.value # Exception instance

I was wondering if there is a reason we wouldn't want to change the code to check for registered unjellyables for the exception, and return them (or rather go through the jelly/unjelly process) if they exist, but return the string representation otherwise. Maybe that is more complicated than I realize.

Any thoughts or suggestions on how we might make this possible? Thanks. -Justin

Submitted Issue844. On Wed, 29 Dec 2004 08:56:48 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

Attached is a patch with modifications to pb.py and jelly.py. With the patch applied, CopyableFailure.getStateToCopy first checks to see if the actual exception instance's class has an unjellyable registered so it can be unjellied on the other side. If it does, the actual exception instance is passed back as failure.value. Otherwise a string representation is passed back as it is today.

I'd like some confirmation that this is an acceptable approach. If so I will gladly submit an enhancement and attach the patch.

-Justin

On Tue, 28 Dec 2004 14:34:02 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

...

Whoops... forgot the attachments. Here they are.

On Tue, 28 Dec 2004 14:32:04 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

...

I would really like to be able to pass custom exceptions over the wire in PB. Previous discussion on this list ended with the conclusion that it currently isn't possible without hacking a local copy of Twisted.

I have done that to get the result I want, with minimal testing, and would like to get some discussion going around this topic. Attached are 3 test files, s.py for the server, c.py for the client, and e.py for the error definitions which are imported by both s.py and c.py. If I make the following modification to my local copy of twisted/spread/pb.py in the CopyableFailure's getStateToCopy method, the custom error is passed back as I want.

#state['value'] = str(self.value) # Exception instance state['value'] = self.value # Exception instance

I was wondering if there is a reason we wouldn't want to change the code to check for registered unjellyables for the exception, and return them (or rather go through the jelly/unjelly process) if they exist, but return the string representation otherwise. Maybe that is more complicated than I realize.

Any thoughts or suggestions on how we might make this possible? Thanks. -Justin

Thanks for the reply. In my testing it seemed that I would get an InsecureJelly error on the sending side if I tried to send back an object that I hadn't called pb.setUnjellyableForClass on the sending side for. In other words, calling setUnjellyableForClass on the sending side was a way of saying that it was okay to send over the wire, and also a way of registering what class to use when unjellying it if it were received. Is this not correct? On 29 Dec 2004 14:15:57 -0500, David Bolen <db3l@fitlinxx.com> wrote:

Justin Johnson <justinjohnson@gmail.com> writes:

...

Attached is a patch with modifications to pb.py and jelly.py. With the patch applied, CopyableFailure.getStateToCopy first checks to see if the actual exception instance's class has an unjellyable registered so it can be unjellied on the other side. If it does, the actual exception instance is passed back as failure.value. Otherwise a string representation is passed back as it is today.

I think the one big risk with this approach is that I'm not sure a sender can ever know accurately whether a recipient will be able to unjelly a particular instance. In your case, you're assuming the two sides are symmetrical and have imported the same definitions, but that need not be the case. At least in general, it's certainly possible for the transmitting side to not have the unjellier registered (if it never expects to receive such objects back, for example). Or conversely, the sender may have an unjellier registered but the recipient won't. In this latter case you'd get an exception on the remote side which sending the string would have avoided.

One alternative approach to handle this is to transmit both the string and instance representation, and then let the decoding side trap any unjellier security exceptions and fall back to the string representation. The big question of course is whether such a fallback should be automatic, silent, etc..

Another option would be to provide some sort of configurability (perhaps something along the lines of how unsafe traceback support is handled) so an application could make the choice of what mode to operate in, either string names, or full instances (and the latter just has to have unjelliers registered just as for any other remotes).

I'm sort of in the same state you are in, in that for me the simplest approach is just transmit the instance because my two sides are in fact symmetric. So the short internal change to local source works fine for me too. But I think that any general addition to the twisted base needs to take the more general proposition (and issues) into account.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Christopher Armstrong <radeex@gmail.com> writes: (...)

Why's that actually a risk? In any case where a PB app uses Copyable and suchlike, that assumption is made. If someone is marking an exception as jellyable, then they know that their clients should also be able to unjelly that exception, and it will be a part of the protocol.

Yeah, that's reasonable. It doesn't cover a desire to transmit system exceptions (e.g., we sometimes reuse ValueError rather than redefining our own version of it), but at worst you'd just need to build a mirror hierarchy for those system exceptions you might reuse. (...)

That's exactly what the patch would do, AIUI. On the server side, you can set unsafeTracebacks if you want tracebacks to be sent to the client. The server side can also set an exception class as jellyable if you want to send exception instances to the client.

Well, the patch is checking the unjelly registry to make the encoding decision on the server side, which I don't think is the same thing as what you're saying. To check for whether the exception was marked as jellyable, it should be checking for a subclass of Jellyable and not looking in the unjelly registry (I think). But yeah, if the patch looked for a Jellyable subclass (which is the same thing the normal _Jellier class does when jellying anything else), I think that would be a reasonable approach. -- David

I agree. I did some testing to verify what you were saying. So this is basically reiterating everything you just said, but just for clarification for myself... in the current non-patched code the following happens: When the server encounters an error while calling a method that was invoked from a remote client, it will... 1) Wrap all subclasses of pb.Error in a pb.CopyableFailure, thus replacing actual pb.Error instances with a string representation of them. 2) Return a new (non-wrapping) pb.CopyableFailure for all other errors, thus including stack trace, etc. When the server needs to send anything else back to the client, if it is a subclass of flavors.Jellyable it sends it back. Otherwise it raises InsecureJelly on the server side. On the client side it will unjelly the thing sent across if it can (i.e. if setUnjellyableForClass was called). Otherwise it raises an InsecureJelly on the client side. So it seems like these are our options: 1) If an error is Jellyable, send it and assume the client will either unjelly it or deal with unjellying errors. 2) If an error is Jellyable, send it. If the client fails to unjelly it, communicate back to the server who then sends a CopyableFailure wrapping the original error. 3) If an error is Jellyable, send both it and a CopyableFailure wrapping the original error. The client can then first try to unjelly the error and fall back to the CopyableFailure that was passed along. I vote for option 1. Options 2 and 3 seem somewhat hacky to me. -Justin On 29 Dec 2004 21:31:17 -0500, David Bolen <db3l@fitlinxx.com> wrote:

Justin Johnson <justinjohnson@gmail.com> writes:

...

In my testing it seemed that I would get an InsecureJelly error on the sending side if I tried to send back an object that I hadn't called pb.setUnjellyableForClass on the sending side for. In other words, calling setUnjellyableForClass on the sending side was a way of saying that it was okay to send over the wire, and also a way of registering what class to use when unjellying it if it were received.

Is this not correct?

I believe the only purpose of setUnjellyableForClass is to establish what to do for unjellying. While it does also impact the global security options (which do get checked during jellying) the way it stores type information in there is only matched during unjellying, at least in my experience (it adds permission for the type but not the instance).

I believe the only (typical) requirement to support jellying an object is that it be a class that is a jelly.Jellyable subclass (such as any of the remoteable flavors like Copyable, Referenceable, etc...). Nothing else should be needed on the server side.

I just tried a quick tweak to your s/e/c.py modules so that s.py returned an instance of a dummy class defined in e.py instead of an error, and it seems to work fine even if the server side (s) hasn't issued the setUnjellyableForClass call. Without that call, the client will raise the error after receiving the object. And even if the server does issue that call, if the dummy class isn't inheriting from Copyable, the insecure error is generated on the transmitting side.

It is, however, possible to insert additional types/classes/modules into the global security options independent of class inheritance from Jellyable. This is how all the basic Python types are handled, but it can be extended to support your own classes (although I had a problem with extension classes/types, since I originally tried to use this to support mxDateTime objects without modifying jelly).

So for example, if I have my test class "MyObject" in e.py, and instead of setUnjellyableForClass, I use something like "jelly.globalSecurity.allowInstancesOf(MyObject)", then the object will still be successfully sent and received (providing the instance data can also be jellied).

So after writing this, the fact that the security options could get used to permit an instance (that isn't of a subclass of Jellyable) to be encoded might mean that just verifying Jellyable (ala my last response to Christopher) is insufficient, at least technically.

Perhaps the only true way to determine if something is jellyable is to try to jelly it (heck, that's probably more Pythonic anyway), and just handle an exception as a fallback to the string representation.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

A new patch is attached. On Thu, 30 Dec 2004 09:57:49 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

Option 1 is also most like the way things work today for sending anything that isn't an error. The only reason CopyableFailure exists is because you HAVE to send a failure. Whether it is jelly/unjellyable or not, the other end must be informed of errors. But as soon as a developer decides to make their errors copyable, they have in effect agreed that they are responsible for testing the unjellying on the other end.

On Thu, 30 Dec 2004 09:48:17 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

...

I agree. I did some testing to verify what you were saying. So this is basically reiterating everything you just said, but just for clarification for myself... in the current non-patched code the following happens:

When the server encounters an error while calling a method that was invoked from a remote client, it will... 1) Wrap all subclasses of pb.Error in a pb.CopyableFailure, thus replacing actual pb.Error instances with a string representation of them. 2) Return a new (non-wrapping) pb.CopyableFailure for all other errors, thus including stack trace, etc.

When the server needs to send anything else back to the client, if it is a subclass of flavors.Jellyable it sends it back. Otherwise it raises InsecureJelly on the server side.

On the client side it will unjelly the thing sent across if it can (i.e. if setUnjellyableForClass was called). Otherwise it raises an InsecureJelly on the client side.

So it seems like these are our options:

1) If an error is Jellyable, send it and assume the client will either unjelly it or deal with unjellying errors.

2) If an error is Jellyable, send it. If the client fails to unjelly it, communicate back to the server who then sends a CopyableFailure wrapping the original error.

3) If an error is Jellyable, send both it and a CopyableFailure wrapping the original error. The client can then first try to unjelly the error and fall back to the CopyableFailure that was passed along.

I vote for option 1. Options 2 and 3 seem somewhat hacky to me.

-Justin

On 29 Dec 2004 21:31:17 -0500, David Bolen <db3l@fitlinxx.com> wrote:

...

Justin Johnson <justinjohnson@gmail.com> writes:

...

In my testing it seemed that I would get an InsecureJelly error on the sending side if I tried to send back an object that I hadn't called pb.setUnjellyableForClass on the sending side for. In other words, calling setUnjellyableForClass on the sending side was a way of saying that it was okay to send over the wire, and also a way of registering what class to use when unjellying it if it were received.

Is this not correct?

I believe the only purpose of setUnjellyableForClass is to establish what to do for unjellying. While it does also impact the global security options (which do get checked during jellying) the way it stores type information in there is only matched during unjellying, at least in my experience (it adds permission for the type but not the instance).

I believe the only (typical) requirement to support jellying an object is that it be a class that is a jelly.Jellyable subclass (such as any of the remoteable flavors like Copyable, Referenceable, etc...). Nothing else should be needed on the server side.

I just tried a quick tweak to your s/e/c.py modules so that s.py returned an instance of a dummy class defined in e.py instead of an error, and it seems to work fine even if the server side (s) hasn't issued the setUnjellyableForClass call. Without that call, the client will raise the error after receiving the object. And even if the server does issue that call, if the dummy class isn't inheriting from Copyable, the insecure error is generated on the transmitting side.

It is, however, possible to insert additional types/classes/modules into the global security options independent of class inheritance from Jellyable. This is how all the basic Python types are handled, but it can be extended to support your own classes (although I had a problem with extension classes/types, since I originally tried to use this to support mxDateTime objects without modifying jelly).

So for example, if I have my test class "MyObject" in e.py, and instead of setUnjellyableForClass, I use something like "jelly.globalSecurity.allowInstancesOf(MyObject)", then the object will still be successfully sent and received (providing the instance data can also be jellied).

So after writing this, the fact that the security options could get used to permit an instance (that isn't of a subclass of Jellyable) to be encoded might mean that just verifying Jellyable (ala my last response to Christopher) is insufficient, at least technically.

Perhaps the only true way to determine if something is jellyable is to try to jelly it (heck, that's probably more Pythonic anyway), and just handle an exception as a fallback to the string representation.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Oh yeah, right. I managed to entirely skip your comments on the security options. How about the attached patch? On 30 Dec 2004 11:52:17 -0500, David Bolen <db3l@fitlinxx.com> wrote:

Justin Johnson <justinjohnson@gmail.com> writes:

...

A new patch is attached.

If we're going with the instance check, I'd probably stick with Jellyable and not Copyable (since you might have a referenceable or viewable exception object) - that matches the actual jelly() implementation.

I'm still wondering if just trying to jelly the object isn't the best approach in the end though, since checking the instance class excludes the cases where the security options permits the object to be sent (e.g., I might use the security options to let standard Python exceptions through without having my own Copyable subclasses).

...

From a prior post of yours:

...

Option 1 is also most like the way things work today for sending anything that isn't an error. The only reason CopyableFailure exists

Close - option 1 (Jellyable subclass) is one half of the jellying process - the other half is the type being allowed by the security options and being supported by the default jelly handling of native Python types.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

See comments below. On 07 Jan 2005 16:20:08 -0500, David Bolen <db3l@fitlinxx.com> wrote: [snip]

Sorry on the delay - holidays and then I forgot about it :-(

I'm still on the fence with respect to having independent "jellyable" checks. My concern would be that having the independent checks of what is suitable for jellying can be fragile with respect to coupling with the actual jelly operation. Even with the jellyable and security checks there are some paths in the jellier code that could still fail the operation.

Can you clarify here? What paths are you referring to? I'd prefer to not duplicate those checks as well, but it seemed like the simplist way to accomplish my goal. It does sound like your understanding of the code is better than mine though.

I also think I liked your earlier attempts at keeping the change within the CopyableFailure class (I may have missed why that changed), although even there you'd have a double jelly since you'd have to check the jelly attempt in getStateToCopy. Although alternatively, CopyableFailure could just implement its own jellyFor (rather than getStateToCopy) which handled that more gracefully and with a single jelly attempt on the value.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Mosas wrote:

Dear All,

I have doubt when I installed twisted web. After the installation twisted web Whether we need any other webserver or not to run web programs? I normally run all the server side coding(perl scripts,php scripts) the following path. "http://192.168.0.21/cgi-bin/helloworld.pl" in my machine. like that How I need to run a perl program in Twisted.

Try: http://twistedmatrix.com/documents/current/howto/using-twistedweb#auto20

I have confusion to give path details for mktap. How I need to give path for mktap to run like (http://192.168.0.21/cgi-bin/helloworld.pl).

Save the above snippet to say www/perlsite.rpy and change the paths to be right for you. Then run

mktap web --resource-script=www/perlsite.rpy twistd -f ./web.tap

Never tried this myself, so .. hoping it works :) good luck, Andy.

On Fri, 11 Mar 2005 08:01:29 -0600, Justin Johnson <justinjohnson@gmail.com> wrote:

Dave,

I mean David. :-)

Okay, I'll see if I can get anyone to commit. On 14 Mar 2005 20:00:14 -0500, David Bolen <db3l@fitlinxx.com> wrote:

Justin Johnson <justinjohnson@gmail.com> writes:

...

Can you comment again on what you think should be done here? I'd like to get this functionality into 2.0 if possible. If you're okay with my patch than I will try to get the tests written before the release. Otherwise I'm wondering if there is an alternative implementation that can be added before the release. I'd prefer not having to maintain local mods to Twisted.

Sorry - I'm pretty sure when I looked at things again, there were not in fact, any paths not covered by your patch. Having the support in the CopyableFailure class versus the jellying code is still debatable, but not enough that I have anything against the current patch.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python