[Twisted-Python] pb.Cacheable doc question
First, the "PB Copyable: Passing Complex Types" doc is *great* and the examples are excellent -- my compliments to all who contributed! My question is about the pb.Cacheable section (http://twistedmatrix.com/documents/current/core/howto/pb-copyable.html#auto9) -- specifically the first sentence: 'Sometimes the object you want to send to the remote process is big and slow. "big" means it takes a lot of data (storage, network bandwidth, processing) to represent its state. "slow" means that state doesn't change very frequently.' I would think that the product of its size and its rate of change is the applicable metric -- i.e.: the bigger the object is *or* the faster it changes (not the slower), the more applicable Cacheable is, no? Steve
On 5 Oct, 08:09 pm, stephen.c.waterbury@nasa.gov wrote:
First, the "PB Copyable: Passing Complex Types" doc is *great* and the examples are excellent -- my compliments to all who contributed!
My question is about the pb.Cacheable section (http://twistedmatrix.com/documents/current/core/howto/pb- copyable.html#auto9) -- specifically the first sentence: 'Sometimes the object you want to send to the remote process is big and slow. "big" means it takes a lot of data (storage, network bandwidth, processing) to represent its state. "slow" means that state doesn't change very frequently.'
I would think that the product of its size and its rate of change is the applicable metric -- i.e.: the bigger the object is *or* the faster it changes (not the slower), the more applicable Cacheable is, no?
That seems plausible. I wonder if the rate comment is motivated by something else, like the chance of the remote cache being out of date when the remote side wants to use some of its data. This would increase with the rate of change, but I don't know if it really matters. I haven't ever actually used a Cacheable myself, as far as I can recall. Jean-Paul
On 10/08/2010 09:25 AM, exarkun@twistedmatrix.com wrote:
On 5 Oct, 08:09 pm, stephen.c.waterbury@nasa.gov wrote:
First, the "PB Copyable: Passing Complex Types" doc is *great* and the examples are excellent -- my compliments to all who contributed!
My question is about the pb.Cacheable section (http://twistedmatrix.com/documents/current/core/howto/pb- copyable.html#auto9) -- specifically the first sentence: 'Sometimes the object you want to send to the remote process is big and slow. "big" means it takes a lot of data (storage, network bandwidth, processing) to represent its state. "slow" means that state doesn't change very frequently.'
I would think that the product of its size and its rate of change is the applicable metric -- i.e.: the bigger the object is *or* the faster it changes (not the slower), the more applicable Cacheable is, no?
That seems plausible. I wonder if the rate comment is motivated by something else, like the chance of the remote cache being out of date when the remote side wants to use some of its data. This would increase with the rate of change, but I don't know if it really matters. I haven't ever actually used a Cacheable myself, as far as I can recall.
Thanks, jp! Fair enough, but that seems equally applicable to Copyable -- a copy could as easily be stale as a cache, I'd think. (Maybe more easily, if it takes longer to update.) So you've just used Copyable, not Cacheable? I wonder if anyone has used it ... maybe I'll be the first! Ooooo. :) Cheers, Steve
On Oct 8, 2010, at 9:25 AM, exarkun@twistedmatrix.com wrote:
On 5 Oct, 08:09 pm, stephen.c.waterbury@nasa.gov wrote:
First, the "PB Copyable: Passing Complex Types" doc is *great* and the examples are excellent -- my compliments to all who contributed!
My question is about the pb.Cacheable section (http://twistedmatrix.com/documents/current/core/howto/pb- copyable.html#auto9) -- specifically the first sentence: 'Sometimes the object you want to send to the remote process is big and slow. "big" means it takes a lot of data (storage, network bandwidth, processing) to represent its state. "slow" means that state doesn't change very frequently.'
I would think that the product of its size and its rate of change is the applicable metric -- i.e.: the bigger the object is *or* the faster it changes (not the slower), the more applicable Cacheable is, no?
That seems plausible. I wonder if the rate comment is motivated by something else, like the chance of the remote cache being out of date when the remote side wants to use some of its data. This would increase with the rate of change, but I don't know if it really matters. I haven't ever actually used a Cacheable myself, as far as I can recall.
I think I probably wrote that paragraph, and it was not very well put. Big objects which are "fast", i.e. change constantly, are perfectly suitable for Cacheables. The point I believe I was trying to make there was that if a significant proportion of the object's data is changing quickly, Cacheable doesn't make much of a difference over just re-Copyable-ing the whole object, since the delta updates will be the same size as the whole object.
On 10/08/2010 03:46 PM, Glyph Lefkowitz wrote:
On Oct 8, 2010, at 9:25 AM, exarkun@twistedmatrix.com <mailto:exarkun@twistedmatrix.com> wrote:
On 5 Oct, 08:09 pm, stephen.c.waterbury@nasa.gov <mailto:stephen.c.waterbury@nasa.gov> wrote:
First, the "PB Copyable: Passing Complex Types" doc is *great* and the examples are excellent -- my compliments to all who contributed!
My question is about the pb.Cacheable section (http://twistedmatrix.com/documents/current/core/howto/pb- copyable.html#auto9) -- specifically the first sentence: 'Sometimes the object you want to send to the remote process is big and slow. "big" means it takes a lot of data (storage, network bandwidth, processing) to represent its state. "slow" means that state doesn't change very frequently.'
I would think that the product of its size and its rate of change is the applicable metric -- i.e.: the bigger the object is *or* the faster it changes (not the slower), the more applicable Cacheable is, no?
That seems plausible. I wonder if the rate comment is motivated by something else, like the chance of the remote cache being out of date when the remote side wants to use some of its data. This would increase with the rate of change, but I don't know if it really matters. I haven't ever actually used a Cacheable myself, as far as I can recall.
I think I probably wrote that paragraph, and it was not very well put. Big objects which are "fast", i.e. change constantly, are perfectly suitable for Cacheables.
The point I believe I was trying to make there was that if /a significant proportion/ of the object's data is changing quickly, Cacheable doesn't make much of a difference over just re-Copyable-ing the whole object, since the delta updates will be the same size as the whole object.
Certainly true. Again, the documentation and the examples are great, anyway, so that part is not a big deal, I was just curious. Thanks, Glyph. Cheers, Steve
Hi, I'm running into crashes with deamonize() from _twistd_unix.py on OSX 10.6 "snow leopard". Crash report contains the ominous flag: USING_FORK_WITHOUT_EXEC_IS_NOT_SUPPORTED_BY_FILE_MANAGER This flag shows up in some issues reported in other python projects on OSX. Apparently 10.5 was more forgiving about this but 10.6 aborts the process. - Has this been an issue for twisted users on OSX 10.6? - Best practices for addressing it? The code calls fork(), but no exec(): def daemonize(): # See http://www.erlenstar.demon.co.uk/unix/faq_toc.html#TOC16 if os.fork(): # launch child and... os._exit(0) # kill off parent os.setsid() if os.fork(): # launch child and... os._exit(0) # kill off parent again. null = os.open('/dev/null', os.O_RDWR) for i in range(3): try: os.dup2(null, i) except OSError, e: if e.errno != errno.EBADF: raise os.close(null) Thanks, Erik van Blokland
On 13/10/10 15:55, Erik van Blokland wrote:
The code calls fork(), but no exec():
fork() with no exec() is legal Unix. Various googling led me to: http://trac.adium.im/ticket/13976#comment:28 """The problem is that OSX since version 10.6 (Snow Leopard) does NOT allow processes launched via Dock (not via exec) to use fork() function, which is one used by Gadu-Gadu resolver. And stuff that crashes is NOT main Adium process, but resolver GG process, that is killed by OS itself. """ If that's the case, I don't think twistd will work in that situation. twistd expects to be able to fork() (unless you give the "-n" argument to keep it in the foreground)
On 13/10/10 16:40, Phil Mayers wrote:
On 13/10/10 15:55, Erik van Blokland wrote:
The code calls fork(), but no exec():
fork() with no exec() is legal Unix.
Various googling led me to:
http://trac.adium.im/ticket/13976#comment:28
"""The problem is that OSX since version 10.6 (Snow Leopard) does NOT allow processes launched via Dock (not via exec) to use fork() function, which is one used by Gadu-Gadu resolver. And stuff that crashes is NOT main Adium process, but resolver GG process, that is killed by OS itself. """
If that's the case, I don't think twistd will work in that situation. twistd expects to be able to fork() (unless you give the "-n" argument to keep it in the foreground)
Other info indicates this is Apple trying to be clever and "protect" applications from some signal handling issues surrounding fork() with libraries which aren't safe in that situation; presumably these are MacOS-specific libraries? If this really is true, then MacOS X is no longer posix-compliant in those circumstances, and Twisted is going to need some work :o( Can you share more details about the application; the context in which it is started, and which system libraries it's likely to load?
On 13/10/10 16:52, Phil Mayers wrote:
If this really is true, then MacOS X is no longer posix-compliant in those circumstances, and Twisted is going to need some work :o(
Can you share more details about the application; the context in which it is started, and which system libraries it's likely to load?
Ah; apparently: http://developer.apple.com/library/ios/#documentation/cocoa/conceptual/Multi... """using the fork function [...] Applications that depend on the Core Foundation, Cocoa, or Core Data frameworks (either explicitly or implicitly) must make a subsequent call to an exec function or those frameworks may behave improperly.""" Bah.
On Oct 13, 2010, at 5:55 PM, Phil Mayers wrote:
Ah; apparently:
http://developer.apple.com/library/ios/#documentation/cocoa/conceptual/Multi...
"""using the fork function [...] Applications that depend on the Core Foundation, Cocoa, or Core Data frameworks (either explicitly or implicitly) must make a subsequent call to an exec function or those frameworks may behave improperly."""
Bah.
Indeed. I presume this is not easily compatible with the way daemonize works right now? Erik
On 13/10/10 16:52, Phil Mayers wrote:
If this really is true, then MacOS X is no longer posix-compliant in those circumstances, and Twisted is going to need some work :o(
One final note (sorry for the fragmentary posts!) - it seems the only places Twisted calls fork() are in BaseProcess (just before an exec() so that's fine), in a couple of test cases, and in the twistd daemonize function. I think you'll just have to add the "-n" argument, or forego twistd. HTH
On Oct 13, 2010, at 5:52 PM, Phil Mayers wrote:
Other info indicates this is Apple trying to be clever and "protect" applications from some signal handling issues surrounding fork() with libraries which aren't safe in that situation; presumably these are MacOS-specific libraries?
If this really is true, then MacOS X is no longer posix-compliant in those circumstances, and Twisted is going to need some work :o(
Can you share more details about the application; the context in which it is started, and which system libraries it's likely to load?
Here's a compact version. The script uses OSX' Quartz to create and manipulate images. Twisted is used to wrap it as a webserver, images are served. More pointers on how to start and use it in the script itself. The real application is more complex, it renders images for sites like this: http://lettersetter.net/ Zip contains 1 test image, 1 script and a sample result image. Running in no-daemon mode (the -n option) prevents the crash, but causes other issues. Thanks! Erik
On Oct 13, 2010, at 12:05 PM, Erik van Blokland wrote:
On Oct 13, 2010, at 5:52 PM, Phil Mayers wrote:
Other info indicates this is Apple trying to be clever and "protect" applications from some signal handling issues surrounding fork() with libraries which aren't safe in that situation; presumably these are MacOS-specific libraries?
If this really is true, then MacOS X is no longer posix-compliant in those circumstances, and Twisted is going to need some work :o(
Can you share more details about the application; the context in which it is started, and which system libraries it's likely to load?
Here's a compact version. The script uses OSX' Quartz to create and manipulate images. Twisted is used to wrap it as a webserver, images are served. More pointers on how to start and use it in the script itself.
You need to avoid using or importing any OSX APIs until after the daemonization has occurred. Unfortunately, twisted executes the entire script file before daemonizing. [that's unfortunate for other reasons besides this, too] Here's a corrected version of your script which works properly. It defers importing Quartz until the reactor is running, by moving it into a function called by reactor.callWhenRunning().
James, On Oct 13, 2010, at 7:20 PM, James Y Knight wrote:
You need to avoid using or importing any OSX APIs until after the daemonization has occurred. Unfortunately, twisted executes the entire script file before daemonizing. [that's unfortunate for other reasons besides this, too]
Here's a corrected version of your script which works properly. It defers importing Quartz until the reactor is running, by moving it into a function called by reactor.callWhenRunning().
Wow! That is incredible. THANKS! Erik
On Oct 13, 2010, at 1:20 PM, James Y Knight wrote:
Here's a compact version. The script uses OSX' Quartz to create and manipulate images. Twisted is used to wrap it as a webserver, images are served. More pointers on how to start and use it in the script itself.
You need to avoid using or importing any OSX APIs until after the daemonization has occurred. Unfortunately, twisted executes the entire script file before daemonizing. [that's unfortunate for other reasons besides this, too]
To the extent that this is an intentional decision (it's at least half accidents of implementation) it's an attempt to allow twistd to actually display an error on the console if there is a serious error starting up, i.e. your plugin is not syntactically valid Python and it's just going to exit. While I remember agreeing that it's unfortunate, I can't actually recall any issues with executing the script before forking. Do we have tickets for them? Can you describe a couple? If we are going to change this behavior, we should make twistd do something more correct, and optionally allow it to report startup errors to the console after it's forked, even if those errors don't necessarily crash it during startup. This isn't rocket science, inheriting file descriptors and delaying exit are all pretty easy.
Here's a corrected version of your script which works properly. It defers importing Quartz until the reactor is running, by moving it into a function called by reactor.callWhenRunning().
Thanks for this, I'm sure that other folks will find it useful. I'm a bit surprised it's not more of a FAQ!
On 08:46 pm, glyph@twistedmatrix.com wrote:
On Oct 13, 2010, at 1:20 PM, James Y Knight wrote:
Here's a compact version. The script uses OSX' Quartz to create and manipulate images. Twisted is used to wrap it as a webserver, images are served. More pointers on how to start and use it in the script itself.
You need to avoid using or importing any OSX APIs until after the daemonization has occurred. Unfortunately, twisted executes the entire script file before daemonizing. [that's unfortunate for other reasons besides this, too]
If we are going to change this behavior, we should make twistd do something more correct, and optionally allow it to report startup errors to the console after it's forked, even if those errors don't necessarily crash it during startup. This isn't rocket science, inheriting file descriptors and delaying exit are all pretty easy.
http://tm.tl/823 Jean-Paul
On Wednesday 13 October 2010, Glyph Lefkowitz wrote:
To the extent that this is an intentional decision (it's at least half accidents of implementation) it's an attempt to allow twistd to actually display an error on the console if there is a serious error starting up, i.e. your plugin is not syntactically valid Python and it's just going to exit.
While I remember agreeing that it's unfortunate, I can't actually recall any issues with executing the script before forking. Do we have tickets for them? Can you describe a couple?
A long time ago I did run into trouble with the uid/gid being applied by twistd after the script is executed: log files would be owned by root instead of by the specified uid/gid. This is not what you asked, but I guess it's also a consequence of the "execute the script first" behavior of twistd.
If we are going to change this behavior, we should make twistd do something more correct, and optionally allow it to report startup errors to the console after it's forked, even if those errors don't necessarily crash it during startup. This isn't rocket science, inheriting file descriptors and delaying exit are all pretty easy.
When using twistd from a typical Linux init script, forking first would lead to the script reporting the startup was a success and after that the error message would be printed. Bye, Maarten
Hi, A while back on Oct 13, 2010, at 7:20 PM, James Y Knight wrote:
You need to avoid using or importing any OSX APIs until after the daemonization has occurred. Unfortunately, twisted executes the entire script file before daemonizing. [that's unfortunate for other reasons besides this, too]
Here's a corrected version of your script which works properly. It defers importing Quartz until the reactor is running, by moving it into a function called by reactor.callWhenRunning().
Deferring OSX API's until after daemonisation was the solution. My test script, fixed by James, works on macbook, but when I run it on a remote OSX server mini, it doesn't. It crashes as before with USING_FORK_WITHOUT_EXEC_IS_NOT_SUPPORTED_BY_FILE_MANAGER) . Both machines run the same versions of python, pyobjc, twisted. Some digging showed that on the mini twisted.application.reactors imports zope.interface, which in turn runs this: __import__('pkg_resources').declare_namespace(__name__) and this causes Carbon to be loaded, before daemonisation. I can't defer loading reactor because I need it to make things tick, so I'm stuck. Could this be related to http://twistedmatrix.com/trac/ticket/4644 ? Any clues to why pkg_resources would cause Carbon to load on one machine, but not on another? Any clues to prevent it these modules from loading? Local: System Version: Mac OS X 10.6.4 (10F569) Kernel Version: Darwin 10.4.0 Remote: System Version: Mac OS X Server 10.6.4 (10F569) Kernel Version: Darwin 10.4.0 Thanks, Erik
On Nov 5, 2010, at 11:46 PM, Erik van Blokland wrote:
Any clues to why pkg_resources would cause Carbon to load on one machine, but not on another? Any clues to prevent it these modules from loading?
FWIW, I found a workaround. The remote machine had setuptools-0.6c11-py2.6.egg, the local machine setuptools-0.6c9-py2.6.egg. Reverting to c9 prevents Carbon to be loaded with pkg_resources, which in turn enables daemonisation. The child happily runs twisted and quartz. I'd still be interested to see how twisted / daemon / OSX API is supposed to work with setuptools c11. But I'm happy I can continue. thanks, Erik
On Nov 6, 2010, at 7:38 AM, Erik van Blokland wrote:
On Nov 5, 2010, at 11:46 PM, Erik van Blokland wrote:
Any clues to why pkg_resources would cause Carbon to load on one machine, but not on another? Any clues to prevent it these modules from loading?
FWIW, I found a workaround. The remote machine had setuptools-0.6c11-py2.6.egg, the local machine setuptools-0.6c9-py2.6.egg. Reverting to c9 prevents Carbon to be loaded with pkg_resources, which in turn enables daemonisation. The child happily runs twisted and quartz.
I'd still be interested to see how twisted / daemon / OSX API is supposed to work with setuptools c11. But I'm happy I can continue.
Well, clearly it doesn't work :). Please feel free to open another ticket. You are correct in saying that this is related to <http://tm.tl/4644>, but that ticket is specifically about a test, and the proposed solution there is to make the test sensitive. <http://bugs.python.org/issue7895> was the bug in Python that causes this, which has since been fixed. Although apparently the fix was not quite in time for 2.7, it looks to me like it has been committed to the 2.7 (and 2.6) branches, so perhaps we will see some point-releases which fix it. So in future versions of Python, Twisted will not be affected by this issue. However, it may be good to get a workaround in Twisted anyway, for those who have to deal with the combination of older Python / newer Setuptools, or a better fix, like changing the order of daemonization. So a ticket is definitely worthwhile.
participants (8)
-
Erik van Blokland -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
James Y Knight -
Maarten ter Huurne -
Phil Mayers -
Stephen Waterbury -
Stephen Waterbury