[Twisted-Python] Dropin.cache errors after clean install of twisted-8.2.0
I recently installed python-2.6.1 and then twisted-8.2.0 (outside of the python tree). One of my users is complaining that when he uses trial to test some twisted programs, although his test passes, he has to stare at a stack trace like this which mentions dropin.cache.new failing to be written: $ cat foo.py #!/sw/external/python-2.6.1/bin/python from twisted.trial import unittest class MyTestClass(unittest.TestCase): def test_foobar(self): print 'hello, world!' self.failUnless(1 == 1) $ /sw/external/twisted-8.2.0/bin/trial ./foo.py /sw/external/twisted-8.2.0/lib/python/twisted/persisted/sob.py:12: DeprecationWarning: the md5 module is deprecated; use hashlib instead import os, md5, sys /sw/external/twisted-8.2.0/lib/python/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha /sw/external/twisted-8.2.0/lib/python/twisted/web/microdom.py:181: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (oldChild.parentNode is self, Traceback (most recent call last): File "/sw/external/twisted-8.2.0/lib/python/twisted/python/usage.py", line 241, in parseOptions self.postOptions() File "/sw/external/twisted-8.2.0/lib/python/twisted/scripts/trial.py", line 293, in postOptions self['reporter'] = self._loadReporterByName(self['reporter']) File "/sw/external/twisted-8.2.0/lib/python/twisted/scripts/trial.py", line 279, in _loadReporterByName for p in plugin.getPlugins(itrial.IReporter): File "/sw/external/twisted-8.2.0/lib/python/twisted/plugin.py", line 200, in getPlugins allDropins = getCache(package) --- <exception caught here> --- File "/sw/external/twisted-8.2.0/lib/python/twisted/plugin.py", line 179, in getCache dropinPath.setContent(pickle.dumps(dropinDotCache)) File "/sw/external/twisted-8.2.0/lib/python/twisted/python/filepath.py", line 623, in setContent f = sib.open('w') File "/sw/external/twisted-8.2.0/lib/python/twisted/python/filepath.py", line 433, in open return open(self.path, mode+'b') exceptions.IOError: [Errno 30] Read-only file system: '/sw/external/twisted-8.2.0/lib/python/twisted/plugins/dropin.cache.new' foo MyTestClass test_foobar ... hello, world! [OK] ------------------------------------------------------------------------ ------- Ran 1 tests in 0.023s PASSED (successes=1) I would consider this to be an install bug. Evidently, trial uses plugins, which upon reading http://twistedmatrix.com/projects/core/documentation/howto/plugin.html#a uto3 I find that plugins need up-to-date dropin.cache files to be built, or else they'll be built dynamically; the latter step fails in my case due to my site-wide install being on a read-only directory. The docs say "when installing or removing software which provides Twisted plugins, the site administrator should be sure the cache is regenerated"; however all I did was install twisted itself, not any specific plugins. All I did to install twisted was 'python setup.py install --home=...' and at no later point did I ever add new plugins to it. Why can't the install take care of initial dropin.cache generation for me? I see a related issue http://twistedmatrix.com/trac/ticket/2409. Regardless of whether ticket 2409 goes anywhere, should I raise an issue that the install should really take care of initial dropin.cache generation, much like python install takes care of std lib .py->.pyc generation? Thanks.
Hi, Just type "sudo twistd" and the errors will go away. (You may already know of this work-around) I agree with you that it would be nice for these to never show up in the first place. I looked at the ticket you mention, but it seems that the cleaner solution would be to resolve these issues at install/upgrade time. -Alex On Wed, Jun 17, 2009 at 12:47 PM, Rutt, Benjamin<Benjamin.Rutt@gs.com> wrote:
-- Alex Clemesha clemesha.org
On Wed, Jun 17, 2009 at 1:41 PM, Alex Clemesha<clemesha@gmail.com> wrote:
Just to follow up on this a bit more, reading from here: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html in the section "Plugin Caching", there is this quote: """ [snip]... when installing or removing software which provides Twisted plugins, the site administrator should be sure the cache is regenerated. Well-behaved package managers for such software should take this task upon themselves, since it is trivially automatable. """ The way I read this, requiring the site admin to run "sudo twistd" to regenerate the cache file is a feature, not a bug. Twisted devs: is this the case? Still maybe the install should do the "trivially automatable" bit? I don't have a strong opinion either way, as I'm just now learning about the details of all this. thanks, Alex
-- Alex Clemesha clemesha.org
On Wed, 17 Jun 2009 14:37:01 -0700, Alex Clemesha <clemesha@gmail.com> wrote:
The "package managers" referred to in that document are things like dpkg and rpm which easily support running arbitrary code after installation. It's rather difficult to automate with distutils, though (which is how the original poster installed Twisted, I am assuming). However, if someone figures out how, a patch will be accepted happily. :) Jean-Paul
Yes, I installed originally with distutils, i.e. 'python setup.py install --home=...' -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jean-Paul Calderone Sent: Wednesday, June 17, 2009 8:57 PM To: Twisted general discussion Subject: Re: [Twisted-Python] Dropin.cache errors after clean install of twisted-8.2.0 On Wed, 17 Jun 2009 14:37:01 -0700, Alex Clemesha <clemesha@gmail.com> wrote:
The "package managers" referred to in that document are things like dpkg and rpm which easily support running arbitrary code after installation. It's rather difficult to automate with distutils, though (which is how the original poster installed Twisted, I am assuming). However, if someone figures out how, a patch will be accepted happily. :) Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I don't have root, so 'sudo twistd' won't help. And as I said, it's a read-only filesystem I was testing on; root wouldn't help anyway. But I could redo my twisted install, and after typing 'python setup.py install --home=...' try just running twistd one time as you suggest, or do python -c 'from twisted.plugin import IPlugin, getPlugins; list(getPlugins(IPlugin))' as the docs suggest. -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Alex Clemesha Sent: Wednesday, June 17, 2009 4:42 PM To: Twisted general discussion Subject: Re: [Twisted-Python] Dropin.cache errors after clean install oftwisted-8.2.0 Hi, Just type "sudo twistd" and the errors will go away. (You may already know of this work-around) I agree with you that it would be nice for these to never show up in the first place. I looked at the ticket you mention, but it seems that the cleaner solution would be to resolve these issues at install/upgrade time. -Alex
On 17 Jun, 07:47 pm, benjamin.rutt@gs.com wrote:
How did you install Twisted? What distribution are you using?
I would consider this to be an install bug. Evidently, trial uses plugins, (...)
I would also consider this to be an installer bug.
No need to open a new ticket. Here's the one you're looking for: http://twistedmatrix.com/trac/ticket/2410 It would be very helpful if you would add a patch for this ticket, though; I'll go so far as to personally volunteer to review it until it's done if you do. However, that ticket is somewhat anemic in its description of the problem. Please be aware that distutils doesn't support post-install hooks in a general way (it leaves things up to the various 'bdist_' plugins), so it's hard to do this in a way which is unambiguously correct. For example, the way post-installation scripts are specified to bdist_wininst, bdist_rpm, and bdist_msi are all subtly different. Also, the debian and redhat packages for twisted use 'setup.py install' but install to a different root, and then have their own custom post- installation hooks. I'm sure that other packages (gentoo, for example) probably invoke setup.py as well, in slightly different ways. bdist_egg doesn't allow for post-installation hooks *at all*, as you can see from <http://bugs.python.org/setuptools/issue41>. So something that modified the behavior of setup.py install would have to carefully encapsulate its post-install hook and only invoke it if appropriate. If you manage to circumnavigate all this nonsense and produce a nice, simple post-install hook with some unit tests, you will be a Twisted hero :).
I installed twisted on linux (RHEL4), solx86 and solsparc using 'python setup.py install --home=...'. I looked at ticket 2410. I see that you added a thorough summary of the problem recently on that ticket - thank you. While I would like to help if I could, I'm afraid I don't know enough about the python packaging technical details to be of much use. Was not aware setup.py could not run arbitrary code at the end. Thanks for the discussion - I'll work around it as needed via python -c 'from twisted.plugin import IPlugin, getPlugins; list(getPlugins(IPlugin))' as a post-install hook of my own. Until a solution emerges, I suppose the INSTALL file in the root of the distribution could contain this text? * Other As with other Python packages, the standard way of installing from source is: python setup.py install If you install from source you may wish to generate the plugin cache, post-install, via: python -c 'from twisted.plugin import IPlugin, getPlugins; list(getPlugins(IPlugin))' See http://twistedmatrix.com/projects/core/documentation/howto/plugin.html for more details. Although it would complicate the INSTALL file quite a bit for an admittedly minor issue. Thanks. -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of glyph@divmod.com Sent: Thursday, June 18, 2009 1:19 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Dropin.cache errors after clean installof twisted-8.2.0 On 17 Jun, 07:47 pm, benjamin.rutt@gs.com wrote:
How did you install Twisted? What distribution are you using?
I would consider this to be an install bug. Evidently, trial uses plugins, (...)
I would also consider this to be an installer bug.
No need to open a new ticket. Here's the one you're looking for: http://twistedmatrix.com/trac/ticket/2410 It would be very helpful if you would add a patch for this ticket, though; I'll go so far as to personally volunteer to review it until it's done if you do. However, that ticket is somewhat anemic in its description of the problem. Please be aware that distutils doesn't support post-install hooks in a general way (it leaves things up to the various 'bdist_' plugins), so it's hard to do this in a way which is unambiguously correct. For example, the way post-installation scripts are specified to bdist_wininst, bdist_rpm, and bdist_msi are all subtly different. Also, the debian and redhat packages for twisted use 'setup.py install' but install to a different root, and then have their own custom post- installation hooks. I'm sure that other packages (gentoo, for example) probably invoke setup.py as well, in slightly different ways. bdist_egg doesn't allow for post-installation hooks *at all*, as you can see from <http://bugs.python.org/setuptools/issue41>. So something that modified the behavior of setup.py install would have to carefully encapsulate its post-install hook and only invoke it if appropriate. If you manage to circumnavigate all this nonsense and produce a nice, simple post-install hook with some unit tests, you will be a Twisted hero :). _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi, Just type "sudo twistd" and the errors will go away. (You may already know of this work-around) I agree with you that it would be nice for these to never show up in the first place. I looked at the ticket you mention, but it seems that the cleaner solution would be to resolve these issues at install/upgrade time. -Alex On Wed, Jun 17, 2009 at 12:47 PM, Rutt, Benjamin<Benjamin.Rutt@gs.com> wrote:
-- Alex Clemesha clemesha.org
On Wed, Jun 17, 2009 at 1:41 PM, Alex Clemesha<clemesha@gmail.com> wrote:
Just to follow up on this a bit more, reading from here: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html in the section "Plugin Caching", there is this quote: """ [snip]... when installing or removing software which provides Twisted plugins, the site administrator should be sure the cache is regenerated. Well-behaved package managers for such software should take this task upon themselves, since it is trivially automatable. """ The way I read this, requiring the site admin to run "sudo twistd" to regenerate the cache file is a feature, not a bug. Twisted devs: is this the case? Still maybe the install should do the "trivially automatable" bit? I don't have a strong opinion either way, as I'm just now learning about the details of all this. thanks, Alex
-- Alex Clemesha clemesha.org
On Wed, 17 Jun 2009 14:37:01 -0700, Alex Clemesha <clemesha@gmail.com> wrote:
The "package managers" referred to in that document are things like dpkg and rpm which easily support running arbitrary code after installation. It's rather difficult to automate with distutils, though (which is how the original poster installed Twisted, I am assuming). However, if someone figures out how, a patch will be accepted happily. :) Jean-Paul
Yes, I installed originally with distutils, i.e. 'python setup.py install --home=...' -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jean-Paul Calderone Sent: Wednesday, June 17, 2009 8:57 PM To: Twisted general discussion Subject: Re: [Twisted-Python] Dropin.cache errors after clean install of twisted-8.2.0 On Wed, 17 Jun 2009 14:37:01 -0700, Alex Clemesha <clemesha@gmail.com> wrote:
The "package managers" referred to in that document are things like dpkg and rpm which easily support running arbitrary code after installation. It's rather difficult to automate with distutils, though (which is how the original poster installed Twisted, I am assuming). However, if someone figures out how, a patch will be accepted happily. :) Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I don't have root, so 'sudo twistd' won't help. And as I said, it's a read-only filesystem I was testing on; root wouldn't help anyway. But I could redo my twisted install, and after typing 'python setup.py install --home=...' try just running twistd one time as you suggest, or do python -c 'from twisted.plugin import IPlugin, getPlugins; list(getPlugins(IPlugin))' as the docs suggest. -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Alex Clemesha Sent: Wednesday, June 17, 2009 4:42 PM To: Twisted general discussion Subject: Re: [Twisted-Python] Dropin.cache errors after clean install oftwisted-8.2.0 Hi, Just type "sudo twistd" and the errors will go away. (You may already know of this work-around) I agree with you that it would be nice for these to never show up in the first place. I looked at the ticket you mention, but it seems that the cleaner solution would be to resolve these issues at install/upgrade time. -Alex
On 17 Jun, 07:47 pm, benjamin.rutt@gs.com wrote:
How did you install Twisted? What distribution are you using?
I would consider this to be an install bug. Evidently, trial uses plugins, (...)
I would also consider this to be an installer bug.
No need to open a new ticket. Here's the one you're looking for: http://twistedmatrix.com/trac/ticket/2410 It would be very helpful if you would add a patch for this ticket, though; I'll go so far as to personally volunteer to review it until it's done if you do. However, that ticket is somewhat anemic in its description of the problem. Please be aware that distutils doesn't support post-install hooks in a general way (it leaves things up to the various 'bdist_' plugins), so it's hard to do this in a way which is unambiguously correct. For example, the way post-installation scripts are specified to bdist_wininst, bdist_rpm, and bdist_msi are all subtly different. Also, the debian and redhat packages for twisted use 'setup.py install' but install to a different root, and then have their own custom post- installation hooks. I'm sure that other packages (gentoo, for example) probably invoke setup.py as well, in slightly different ways. bdist_egg doesn't allow for post-installation hooks *at all*, as you can see from <http://bugs.python.org/setuptools/issue41>. So something that modified the behavior of setup.py install would have to carefully encapsulate its post-install hook and only invoke it if appropriate. If you manage to circumnavigate all this nonsense and produce a nice, simple post-install hook with some unit tests, you will be a Twisted hero :).
I installed twisted on linux (RHEL4), solx86 and solsparc using 'python setup.py install --home=...'. I looked at ticket 2410. I see that you added a thorough summary of the problem recently on that ticket - thank you. While I would like to help if I could, I'm afraid I don't know enough about the python packaging technical details to be of much use. Was not aware setup.py could not run arbitrary code at the end. Thanks for the discussion - I'll work around it as needed via python -c 'from twisted.plugin import IPlugin, getPlugins; list(getPlugins(IPlugin))' as a post-install hook of my own. Until a solution emerges, I suppose the INSTALL file in the root of the distribution could contain this text? * Other As with other Python packages, the standard way of installing from source is: python setup.py install If you install from source you may wish to generate the plugin cache, post-install, via: python -c 'from twisted.plugin import IPlugin, getPlugins; list(getPlugins(IPlugin))' See http://twistedmatrix.com/projects/core/documentation/howto/plugin.html for more details. Although it would complicate the INSTALL file quite a bit for an admittedly minor issue. Thanks. -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of glyph@divmod.com Sent: Thursday, June 18, 2009 1:19 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Dropin.cache errors after clean installof twisted-8.2.0 On 17 Jun, 07:47 pm, benjamin.rutt@gs.com wrote:
How did you install Twisted? What distribution are you using?
I would consider this to be an install bug. Evidently, trial uses plugins, (...)
I would also consider this to be an installer bug.
No need to open a new ticket. Here's the one you're looking for: http://twistedmatrix.com/trac/ticket/2410 It would be very helpful if you would add a patch for this ticket, though; I'll go so far as to personally volunteer to review it until it's done if you do. However, that ticket is somewhat anemic in its description of the problem. Please be aware that distutils doesn't support post-install hooks in a general way (it leaves things up to the various 'bdist_' plugins), so it's hard to do this in a way which is unambiguously correct. For example, the way post-installation scripts are specified to bdist_wininst, bdist_rpm, and bdist_msi are all subtly different. Also, the debian and redhat packages for twisted use 'setup.py install' but install to a different root, and then have their own custom post- installation hooks. I'm sure that other packages (gentoo, for example) probably invoke setup.py as well, in slightly different ways. bdist_egg doesn't allow for post-installation hooks *at all*, as you can see from <http://bugs.python.org/setuptools/issue41>. So something that modified the behavior of setup.py install would have to carefully encapsulate its post-install hook and only invoke it if appropriate. If you manage to circumnavigate all this nonsense and produce a nice, simple post-install hook with some unit tests, you will be a Twisted hero :). _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (4)
-
Alex Clemesha -
glyph＠divmod.com -
Jean-Paul Calderone -
Rutt, Benjamin