installing .py plugins to an alternate directory
Hi all, I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like: installs to $AKARA/spam_extension.py (note: only .py files are supported; not even .pyc files) ========================= from akara.services import simple_service import my_spam # This is part of the distribution, and gets put in site-packages @simple_service("GET", "http://vikings.protocol.id/") def vikings(say=my_spam.DEFAULT_TEXT): return my_spam.vikings(say) ========================= We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say: from distutils.core import setup from akara.distutils ... I'm not sure what here ... setup(name="Spam services", package="my_spam", akara_extensions=["spam_extension.py"] ) To clarify, the development/distribution package looks like: $PACKAGE/setup.py $PACKAGE/README $PACKAGE/spam_extensions.py $PACKAGE/my_spam/__init__.py $PACKAGE/my_spam/dramatis_personae.py $PACKAGE/my_spam/cafe.py and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages. The installation does not need to byte-compile spam_extension.py. It should also include spam_extension.py in any distribution that it makes. I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us. Suggestions? I had hoped that I could add my own cmdclasses like "build_akara" and "install_akara" which would get called during the correct stages of the build process, but that seems to be a dead end. I might be able to hack the "install_data" cmdclass to make it work, but that would prevent anyone from using it in their own setup.py distributions. Andrew dalke@dalkescientific.com
Andrew Dalke wrote:
Hi all,
I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like:
installs to $AKARA/spam_extension.py (note: only .py files are supported; not even .pyc files) ========================= from akara.services import simple_service
import my_spam # This is part of the distribution, and gets put in site-packages
@simple_service("GET", "http://vikings.protocol.id/") def vikings(say=my_spam.DEFAULT_TEXT): return my_spam.vikings(say) =========================
We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say:
from distutils.core import setup from akara.distutils ... I'm not sure what here ...
setup(name="Spam services", package="my_spam", akara_extensions=["spam_extension.py"] )
To clarify, the development/distribution package looks like: $PACKAGE/setup.py $PACKAGE/README $PACKAGE/spam_extensions.py $PACKAGE/my_spam/__init__.py $PACKAGE/my_spam/dramatis_personae.py $PACKAGE/my_spam/cafe.py
and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages.
The installation does not need to byte-compile spam_extension.py.
It should also include spam_extension.py in any distribution that it makes.
I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us.
I won't comment on entry_points, as I have never used them. The way I would do it is by having akara distutils extensions, which define in particular a setup function and associated classes. Here is the code: import shutil from distutils.core import setup as _setup from distutils.core import Command from distutils import log # XXX: you will need to handle setuptools (and distribute, and....) # monkey-patching for the below commands from distutils.command.install import install as old_install from distutils.dist import Distribution as _Distribution # Where to install akara extensions AKARA_SITE = '/tmp' def setup(**kw): new_kw = kw.copy() # Handle commands overriding cmdclass = new_kw.get('cmdclass', {}) if 'install' in cmdclass: install = cmdclass['install'] else: install = old_install class my_install(install): sub_commands = install.sub_commands + [ ('install_akara_extensions', lambda x: True) ] class install_akara_extensions(Command): description = "Command to install akara extensions" user_options = [ ('akara-site=', None, 'Akara plugin directory install'), ] def initialize_options(self): self.install_dir = None self.akara_site = None self.outfiles = [] def finalize_options(self): if self.akara_site is None: self.akara_site = AKARA_SITE def run (self): dist = self.distribution for a in dist.akara_extensions: log.info("Installing akara extension %s in %s" % (a, self.akara_site)) shutil.copy(a, self.akara_site) new_cmdclass = {} new_cmdclass.update(cmdclass) new_cmdclass['install'] = my_install new_cmdclass['install_akara_extensions'] = install_akara_extensions new_kw['cmdclass'] = new_cmdclass # Handle overriden distclass if not 'distclass' in new_kw: Distribution = new_kw['distclass'] else: Distribution = _Distribution class MyDistribution(Distribution): def __init__(self, attrs=None): assert not hasattr(self, 'akara_extensions') if 'akara_extensions' in attrs: self.akara_extensions = attrs['akara_extensions'] else: self.akara_extensions = [] Distribution.__init__(self, attrs) new_kw['distclass'] = MyDistribution return _setup(**new_kw) setup(name="Spam services", packages=["my_spam"], akara_extensions=["spam_extension.py"] ) This is the minimal code to support this semi correctly. Note that many things are not handled correctly: - get_output for install_akara_extensions should be handled - most likely it will break for semi-random distutils extensions, including setuptools and distribute: you will need to detect monkey-patching to decide which Distribution and install commands to override. I have not done it because it requires the extensions to be in a separate package (to conditionally import things depending on detected monkey patching). If you think this is insane, you are not alone :) David
On Dec 3, 2009, at 4:46 AM, David Cournapeau wrote:
The way I would do it is by having akara distutils extensions, which define in particular a setup function and associated classes.
...
If you think this is insane, you are not alone :)
Wow. That's just crazy, I say - crazy. I considered something like that, but now I see what I was thinking of wouldn't have worked. Okay, looks like I'll have to get my mucking gear on. Andrew dalke@dalkescientific.com
On Dec 3, 2009, at 4:46 AM, David Cournapeau wrote:
If you think this is insane, you are not alone :)
I think that by hooking into "data_files" I can be a bit less insane. What's wrong with this? Other than that it writes Writing /Library/Python/2.6/site-packages/Spam_services-0.0.0-py2.6.egg-info which isn't needed when there is no other Python code. But I can special case that and bypass the setup mechanism entirely if there are no py_modules, packages, or ext_modules given. (Am I missing something?) from distutils.core import setup def my_setup(**kwargs): if "akara_extensions" in kwargs: akara_extensions = kwargs["akara_extensions"] if not isinstance(akara_extensions, list): raise TypeError("akara_extensions must be a list of filenames") del kwargs["akara_extensions"] if "data_files" not in kwargs: kwargs["data_files"] = [] data_files = kwargs["data_files"] data_files.append( ("/tmp", akara_extensions) ) setup(**kwargs) my_setup(name="Spam services", #packages=["my_spam"], akara_extensions = ["spam_extensions.py"], ) Andrew dalke@dalkescientific.com
On Sat, Dec 5, 2009 at 10:27 AM, Andrew Dalke <dalke@dalkescientific.com> wrote:
On Dec 3, 2009, at 4:46 AM, David Cournapeau wrote:
If you think this is insane, you are not alone :)
I think that by hooking into "data_files" I can be a bit less insane. What's wrong with this?
The problem of data_files is that its semantics are heavily changed between the different tools (distutils, setuptools, etc...), so covering all cases is difficult. From my experience, creating a new command makes the thing more robust against monkey patching (don't expect too much, though). David
At 01:06 AM 12/3/2009 +0100, Andrew Dalke wrote:
Hi all,
I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like:
installs to $AKARA/spam_extension.py (note: only .py files are supported; not even .pyc files) ========================= from akara.services import simple_service
import my_spam # This is part of the distribution, and gets put in site-packages
@simple_service("GET", "http://vikings.protocol.id/") def vikings(say=my_spam.DEFAULT_TEXT): return my_spam.vikings(say) =========================
We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say:
from distutils.core import setup from akara.distutils ... I'm not sure what here ...
setup(name="Spam services", package="my_spam", akara_extensions=["spam_extension.py"] )
To clarify, the development/distribution package looks like: $PACKAGE/setup.py $PACKAGE/README $PACKAGE/spam_extensions.py $PACKAGE/my_spam/__init__.py $PACKAGE/my_spam/dramatis_personae.py $PACKAGE/my_spam/cafe.py
and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages.
The installation does not need to byte-compile spam_extension.py.
It should also include spam_extension.py in any distribution that it makes.
I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us.
It wouldn't be so much of a change as an addition. You'd just add code like this, either before or after your existing loop over the extensions directory: for entry_point in pkg_resources.iter_entry_points('akara'): extension_module = entry_point.load() # do stuff with extension_module And then users would declare their extensions for installation like this: setup(name="Spam services", packages=["my_spam"], py_modules=["spam_extension"], entry_points={'akara':'Spam services=spam_extension'} # arbitrary name=module ) Everything else would be the same as you described above with respect to layout, except: 1. the spam_extension module would be installed in site-packages 2. It wouldn't need to be a top-level module (i.e., it could be a module in the package) 3. You don't need any custom distutils extensions, except what you get via setuptools or Distribute.
On Dec 3, 2009, at 7:58 PM, P.J. Eby wrote:
It wouldn't be so much of a change as an addition. You'd just add code like this, either before or after your existing loop over the extensions directory:
for entry_point in pkg_resources.iter_entry_points('akara'): extension_module = entry_point.load() # do stuff with extension_module
I sent a followup to Tarek's reply on some of the architectural changes I meant, like changing from the current ability to identify/enable/disable/rename web services with text editor to one which I don't know much about and which seems more complicated. While thinking about this, there's another one I came up with. We use a pre-forking server based on flup. The master starts up and spawns the processes which do the listening. It's these processes which import the extensions. Send a SIGHUP to the master and it restarts the children, which in turn rescan the list of extensions and reimport them. It appears that pkg_resources does some caching, including using linecache. I can't tell how well it would work if our pluging packages were updated after the main server was running. Andrew dalke@dalkescientific.com
At 01:18 AM 12/5/2009 +0100, Andrew Dalke wrote:
It appears that pkg_resources does some caching, including using linecache. I can't tell how well it would work if our pluging packages were updated after the main server was running.
If you need to ensure that you get a fresh list of plugins each time, you can use "for entry_point in pkg_resources.WorkingSet().iter_entry_points(...)". Or more precisely, you can create a new WorkingSet() whenever you want to start over with a clean cache. (Btw, the only thing pkg_resources uses linecache for is to ensure that the source of a script loaded from an alternate location is viewable in linecache as though it came from the location the script was run from. This is a debugging aid only, and doesn't have any effect on anything that's not trying to dump out source code lines (such as an error reporting tool, or the Python debugger).)
On Thu, Dec 3, 2009 at 1:06 AM, Andrew Dalke <dalke@dalkescientific.com> wrote:
Hi all,
I'm working with the Akara project. It contains a web server. The server loads extensions from a special directory (let's say "$AKARA" for now). An extension can register handlers for URLs. An example extension might look like:
installs to $AKARA/spam_extension.py (note: only .py files are supported; not even .pyc files) ========================= from akara.services import simple_service
import my_spam # This is part of the distribution, and gets put in site-packages
@simple_service("GET", "http://vikings.protocol.id/") def vikings(say=my_spam.DEFAULT_TEXT): return my_spam.vikings(say) =========================
We want people to be able to distribute Akara plugins and install via setup.py. Ideally I would like to say:
from distutils.core import setup from akara.distutils ... I'm not sure what here ...
setup(name="Spam services", package="my_spam", akara_extensions=["spam_extension.py"] )
To clarify, the development/distribution package looks like: $PACKAGE/setup.py $PACKAGE/README $PACKAGE/spam_extensions.py $PACKAGE/my_spam/__init__.py $PACKAGE/my_spam/dramatis_personae.py $PACKAGE/my_spam/cafe.py
and $PACKAGE/spam_extensions.py goes to $AKARA/spam_extensions.py while $PACKAGE/my_spam is copied to site-packages.
The installation does not need to byte-compile spam_extension.py.
It should also include spam_extension.py in any distribution that it makes.
I looked through the documentation and searched for existing examples, but found nothing which does this. The plugins I found used entry_points, and that's an architecture change which I don't think is appropriate for us.
Suggestions?
What about having an explicit configuration file in Akara for plugins, where you just add extensions, exactly like mercurial does: [extensions] foo = package.spam_extension bar = spam_extension2 where "package.spam_extension" and "spam_extension2" are modules Akara would simply __import__() Meaning a plugin will be a normal project that gets installed, and then configured to be used in Akara. Tarek
Tarek Ziadé wrote:
where "package.spam_extension" and "spam_extension2" are modules Akara would simply __import__() Meaning a plugin will be a normal project that gets installed, and then configured to be used in Akara.
This solution is simpler, but it does not solve the issue of installing the plugin outside site-packages. I think it is good practice not to pollute site-packages with app-specific plugins. David
David Cournapeau a écrit :
Tarek Ziadé wrote:
where "package.spam_extension" and "spam_extension2" are modules Akara would simply __import__() Meaning a plugin will be a normal project that gets installed, and then configured to be used in Akara.
This solution is simpler, but it does not solve the issue of installing the plugin outside site-packages. I think it is good practice not to pollute site-packages with app-specific plugins. ZopeComponentArchitecture/Entry points with an assembler like buildout for the later won't pollute site-packages
David _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
On Dec 4, 2009, at 9:18 AM, kiorky wrote:
ZopeComponentArchitecture/Entry points with an assembler like buildout for the later won't pollute site-packages
I tried searching for "ZopeComponentArchitecture/Entry points" but found nothing that seemed relevant. Did you mean "setuptools entry_points"? I only know a bit about them, in the context of using TurboGears, but have not used them in my own code. I had some concerns about them. For one, all of the plugins define out-word facing web services on our server. If the plugins can be located in arbitrary locations inside of site-packages, how does the administrator know which plugins will be activated? How does the admin enable/disable a plugin for testing or security reasons, except by changing the entire package installation? If the admin wants to change the URL for a given service from "/spam" to "/spam_and_eggs", which is currently done as configuration data in the installed plugin, file, that is, by changing # default service name is base on the function name. This becomes "/spam" @simple_service("GET", "http://protocol.id/") def spam(): return "Spam!" -to- @simple_service("GET", "http://protocol.id/", "spam_and_eggs") def spam(): return "Spam!" It does not seem like changing the installed package will be so simple. It would mean searching the installed site-packages to find "spam" (instead of now, which is done with a grep of a single directory) and possibly made more difficult if the package is installed as an egg. These are reasons I decided not to look into setuptools' entry_points as a solution to this problem. Andrew dalke@dalkescientific.com
At 12:34 AM 12/5/2009 +0100, Andrew Dalke wrote:
I had some concerns about them. For one, all of the plugins define out-word facing web services on our server. If the plugins can be located in arbitrary locations inside of site-packages, how does the administrator know which plugins will be activated? How does the admin enable/disable a plugin for testing or security reasons, except by changing the entire package installation?
Then install the plugins as eggs to the plugin directory, rather than installing them in site-packages.
If the admin wants to change the URL for a given service from "/spam" to "/spam_and_eggs", which is currently done as configuration data in the installed plugin, file, that is, by changing
# default service name is base on the function name. This becomes "/spam" @simple_service("GET", "http://protocol.id/") def spam(): return "Spam!"
-to-
@simple_service("GET", "http://protocol.id/", "spam_and_eggs") def spam(): return "Spam!"
It does not seem like changing the installed package will be so simple.
Indeed. It would be much better to make your service decorator read overrides from a configuration file, so that the decorator values are merely defaults. Editing source code is a lousy way to do configuration, when there's stuff in the file besides configuration.
Andrew Dalke a écrit :
On Dec 4, 2009, at 9:18 AM, kiorky wrote:
ZopeComponentArchitecture/Entry points with an assembler like buildout for the later won't pollute site-packages
I tried searching for "ZopeComponentArchitecture/Entry points" but found nothing that seemed relevant. Did you mean "setuptools entry_points"? I mean two approaches:
- setuptools entry points, if you need just simple entry points registered when your plugins are in the sys.path of your package. So when you have generated scripts with buildout, or when you have done some easy_install or python setup.py install dance. - Zope Component Architecture, if you need something more complicated. The later can be tedious at first if you are not yet familiar with zope as you ll surely have to initiate a 'site' yourself and do other stuff around if you are not in a zope environment, but that is just a matter of some readings and hacks. Starters are the zope.interface, zope.configuration, and zope.component pypi pages ([1]) and also this one [2]. Note that Trac do something similar on it's own if i could remember well. [1] - http://pypi.python.org/pypi/zope.configuration http://pypi.python.org/pypi/zope.interface http://pypi.python.org/pypi/zope.component [2] - http://www.muthukadan.net/docs/zca.html
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
David Cournapeau wrote:
This solution is simpler, but it does not solve the issue of installing the plugin outside site-packages. I think it is good practice not to pollute site-packages with app-specific plugins.
Just use buildout and be done with it ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Dec 4, 2009, at 5:29 AM, David Cournapeau wrote:
This solution is simpler, but it does not solve the issue of installing the plugin outside site-packages. I think it is good practice not to pollute site-packages with app-specific plugins.
This tangential topic came up recently because of our recent GothPy (local Gothenburg, Sweden Python User's Group) meeting. One of the members develops TextTest http://texttest.carmen.se/index.php?page=main It is a command-line application with both GUI and non-GUI modes. The GUI uses Gtk. He does not distribute it with setup.py . The installation instructions are "call $texttest/bin/texttest.py" or set your path to include the right directory. This causes a problem with me because my system Python is not configured for PyGtk. I have a Mac. I used Macports to install PyGtk and use the Python in /opt/local/bin/python2.6 . I ended up putting that path in by-hand. I would rather have had the normal setup.py script installation program take care of that for me. However, he does not want to install it as a package because the code is not designed as a library, and he doesn't want to "pollute" site-packages either with this application code. I said it was okay, and that people write command-line programs like this #!/usr/bin/env python import application_package application_package.main() I thought that was common practice, and I don't see it as pollution. But now I see my views aren't as much in the majority as I thought they were. What should the practice be? Put this into "/usr/local/lib/application_package-$version/ and have the command-line program look more like #!/usr/bin/env python import sys sys.path.insert(0, "/usr/local/lib/application-package-$version") import application_package application_package.main() ? Andrew dalke@dalkescientific.com
On Dec 3, 2009, at 8:09 PM, Tarek Ziadé wrote:
What about having an explicit configuration file in Akara for plugins, where you just add extensions, exactly like mercurial does:
[extensions] foo = package.spam_extension bar = spam_extension2
where "package.spam_extension" and "spam_extension2" are modules Akara would simply __import__() Meaning a plugin will be a normal project that gets installed, and then configured to be used in Akara.
Since I don't know how mercurial does this, I'm going to have to guess on your meaning here and reading the hg documentation. There seem to be two ways to install an extension. One is by editing the hgrc file, and the other is by putting the Python code in the "hgext" module. I looked at the installation instructions for mercurial packages listed at http://mercurial.selenic.com/wiki/UsingExtensions . They all require hand-editing of the hgrc file. That is not what I want. What I want is "python setup.py install" to work, and to allow third-party extensions to use the existing Python setup framework to install not only the extension component, but any other packages and even compiled C code which might be needed. The other option was to put files into hgext, but that seems only appropriate for the extensions shipped with Mercurial, and I found nothing which helped automating the installation of those files. So your recommendation does not seem helpful. Andrew dalke@dalkescientific.com
On Sat, Dec 5, 2009 at 12:22 AM, Andrew Dalke <dalke@dalkescientific.com> wrote: [..]
I looked at the installation instructions for mercurial packages listed at http://mercurial.selenic.com/wiki/UsingExtensions . They all require hand-editing of the hgrc file. That is not what I want.
Yes, that's how it works, you can add in ~/.hgrc: [extensions] package.spam_extension = ... And mercurial will enventually do an __import__("package.spam_extension"). That's an explicit plugin system. Setuptools entry_points are a plugin system based on discovery: you register some code and you can iterate on all registered code from your app. In the next mail you are writing about entry_points this:
I had some concerns about them. For one, all of the plugins define out-word facing web services on our server. If the plugins can be located in arbitrary locations inside of site-packages, how does the administrator know which plugins will be activated? How does the admin enable/disable a plugin for testing or security reasons, except by changing the entire package installation?
This means that you want to be able to *manage* and configure your plugins, so unless you are driving them from an explicit central configuration system like mercurial has, you won't be able to fully control plugins, which can lead to the limitations you are describing. The other way is to force the plugins to be located in the same directory as you said, but this means that you want to allow any third-party application to install itself as a plugin in there. Then if the administrator wants to deactivate some plugins, what will he do ? remove the file that was added in the special directory ? That could make that particular plugin half-removed if it has other files in the system. IOW if a plugin is installed in the system, it has to be fully installed within your application if you want full control over it. That's why I was thinking of a configuration file, where it is easy to control your plugins explicitely. Of course, until we have the APIs (PEP 376) to uninstall packages easily, it's not easy to remove an installed project that may contain the plugin, unless it was installed with Pip. Or, maybe you could use self-contained eggs, like zc.buildout does, that might be the only use case for them Regards Tarek
participants (7)
-
Andrew Dalke -
Chris Withers -
David Cournapeau -
David Cournapeau -
kiorky -
P.J. Eby -
Tarek Ziadé