[Twisted-Python] Process completion not detected on HP-UX (#608)
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
I ran into a problem on HP-UX (B.11.23 U ia64) using Python 2.4.3, whereby processes started via reactor.spawnProcess() never seem to get the completion event. I fished around via Google and found that this is ticket #608, which has been hanging around for over 2 years. I'd be willing to help troubleshoot this if someone could provide some guidance. I'd really like to get this fixed ASAP, so any help is appreciated. Thanks. Please reply to me directly in addition to the list, as I'm not subscribed. -Glenn Hochberg
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 11:24:55 -0400, Glenn Hochberg <gah@research.att.com> wrote:
I've commented on #608. Long story short: we need a buildbot, and probably a maintainer for HP/UX. Interested in volunteering? You probably want to add yourself as "cc" specifically on that bug, if you don't want to subscribe to the mailing list.
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
On Jul 31, 2006, at 11:43 AM, glyph@divmod.com wrote:
Actually, I had to subscribe anyway (my email was rejected the first time), Pardon my ignorance, but what is a "buildbot" exactly--obviously something that builds Twisted, but is it something I'd run myself, or something that needs to be accessible from the Internet? I could help out in the short term, but I'm no HP-UX wiz; I'm only using it for one particular project because that's the hardware they run it on (to my dismay). -Glenn
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 12:05:16 -0400, Glenn Hochberg <gah@research.att.com> wrote:
It's the thing driving this page: http://twistedmatrix.com/buildbot/ It's how we can tell if Twisted works on various platforms or not. Without a buildslave running on a particular platform, we have no way to keep track of whether changes being made are correct on that platform. Jean-Paul
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 12:57:48 -0400, Jean-Paul Calderone <exarkun@divmod.com> wrote:
If you're interested in setting up a build slave, http://buildbot.sourceforge.net/README would be a good place to start: see the "setting up a build slave" section. IIRC, you'll need some cooperation from our end as well but that should get you started.
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
On Jul 31, 2006, at 1:51 PM, glyph@divmod.com wrote:
Sadly, I will not be able to set up a build slave, as I do not have a machine that is accessible from the general internet. Also, the HP testdrive machines cannot make outgoing connections, so they would not be able to do checkouts. However, I am still interested in finding a solution to #608. If we can come up with a patch that fixes the problem on HP-UX w/o breaking anything on any other platform, would that be sufficient? You could then test that it does no harm, at least, and I could test whether the fix actually works on HP-UX. What approach would you recommend in troubleshooting the problem--is there built-in debugging that can be turned on that might give me some more detailed information, or do I need to start adding log or print statements? Again, any help would be appreciated. -Glenn
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 15:58:40 -0400, Glenn Hochberg <gah@research.att.com> wrote:
Run one of the unit tests which fails in debug mode. For example, trial -b twisted.trial.test.test_process.ProcessTestCase.testProcess If none of the existing unit tests fail, write a new one :) Jean-Paul
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
After much staring at the Twisted code and some experimenting, I finally discovered the nature of the problem. The semantics of the select() call on HP-UX with respect to pipes is a bit different than the other OS's I have access to (i.e. Linux, Solaris, and Mac OS X). In particular, the "read hack" in process.py does not work, because select shows the write-end of a pipe being both readable and writable whenever there is data in the pipe and the pipe is not full. There does not appear to be any way to detect when all readers of the pipe have closed without actually writing at least one byte to the write- end (in which case you get a broken pipe error, as one would expect). As a temporary workaround, I am doing the following in process.py: a) detecting hp-ux pipe behavior and setting a global brokenHpUxPipeBehavior b) disabling the read hack if this variable is set c) in Process.processEnded() I check for the variable and if set, send closeConnection to each ProcessWriter in the pipes dict This seems to do the trick. I have posted to the comp.unix.hp.hpux newsgroup looking for a way to determine if the readers are all gone, but don't know whether one will turn up or not. Anyone have any suggestions on better ways to fix this? -Glenn On Jul 31, 2006, at 4:36 PM, Jean-Paul Calderone wrote:
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 11:24:55 -0400, Glenn Hochberg <gah@research.att.com> wrote:
I've commented on #608. Long story short: we need a buildbot, and probably a maintainer for HP/UX. Interested in volunteering? You probably want to add yourself as "cc" specifically on that bug, if you don't want to subscribe to the mailing list.
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
On Jul 31, 2006, at 11:43 AM, glyph@divmod.com wrote:
Actually, I had to subscribe anyway (my email was rejected the first time), Pardon my ignorance, but what is a "buildbot" exactly--obviously something that builds Twisted, but is it something I'd run myself, or something that needs to be accessible from the Internet? I could help out in the short term, but I'm no HP-UX wiz; I'm only using it for one particular project because that's the hardware they run it on (to my dismay). -Glenn
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 12:05:16 -0400, Glenn Hochberg <gah@research.att.com> wrote:
It's the thing driving this page: http://twistedmatrix.com/buildbot/ It's how we can tell if Twisted works on various platforms or not. Without a buildslave running on a particular platform, we have no way to keep track of whether changes being made are correct on that platform. Jean-Paul
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 12:57:48 -0400, Jean-Paul Calderone <exarkun@divmod.com> wrote:
If you're interested in setting up a build slave, http://buildbot.sourceforge.net/README would be a good place to start: see the "setting up a build slave" section. IIRC, you'll need some cooperation from our end as well but that should get you started.
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
On Jul 31, 2006, at 1:51 PM, glyph@divmod.com wrote:
Sadly, I will not be able to set up a build slave, as I do not have a machine that is accessible from the general internet. Also, the HP testdrive machines cannot make outgoing connections, so they would not be able to do checkouts. However, I am still interested in finding a solution to #608. If we can come up with a patch that fixes the problem on HP-UX w/o breaking anything on any other platform, would that be sufficient? You could then test that it does no harm, at least, and I could test whether the fix actually works on HP-UX. What approach would you recommend in troubleshooting the problem--is there built-in debugging that can be turned on that might give me some more detailed information, or do I need to start adding log or print statements? Again, any help would be appreciated. -Glenn
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006 15:58:40 -0400, Glenn Hochberg <gah@research.att.com> wrote:
Run one of the unit tests which fails in debug mode. For example, trial -b twisted.trial.test.test_process.ProcessTestCase.testProcess If none of the existing unit tests fail, write a new one :) Jean-Paul
![](https://secure.gravatar.com/avatar/242f326822e54a5819c174628c6020f8.jpg?s=120&d=mm&r=g)
After much staring at the Twisted code and some experimenting, I finally discovered the nature of the problem. The semantics of the select() call on HP-UX with respect to pipes is a bit different than the other OS's I have access to (i.e. Linux, Solaris, and Mac OS X). In particular, the "read hack" in process.py does not work, because select shows the write-end of a pipe being both readable and writable whenever there is data in the pipe and the pipe is not full. There does not appear to be any way to detect when all readers of the pipe have closed without actually writing at least one byte to the write- end (in which case you get a broken pipe error, as one would expect). As a temporary workaround, I am doing the following in process.py: a) detecting hp-ux pipe behavior and setting a global brokenHpUxPipeBehavior b) disabling the read hack if this variable is set c) in Process.processEnded() I check for the variable and if set, send closeConnection to each ProcessWriter in the pipes dict This seems to do the trick. I have posted to the comp.unix.hp.hpux newsgroup looking for a way to determine if the readers are all gone, but don't know whether one will turn up or not. Anyone have any suggestions on better ways to fix this? -Glenn On Jul 31, 2006, at 4:36 PM, Jean-Paul Calderone wrote:
participants (3)
-
Glenn Hochberg
-
glyph@divmod.com
-
Jean-Paul Calderone