virtualenv & pypi-server
After reading everything was possible to read on setuptools and friends, I'm a bit confused...
I have the following situation: suppose that there is a framework composed by many many eggs, which doesn't change very often.
Then there are N applications using this framework, where every application is also composed by 2-3 eggs, and everything is run in a "funny" way (using Envisage).
So at the moment every application fiddles around with the global easy_install.pth, which is bad and needs to be changed, and I have to figure out how.
My idea would be to - have a global pypi-server running on localhost - every application is in a virtualenv (or buildout environment) and uses this local pypi-server
I'm doing a few experiments, but already I'm not able to make my virtual env to fetch from my local pypi-server, what is the magic setting for that?
The other future problems that I still see now are: 1. I have to be able to develop also the eggs from the framework 2. I have to make everything very simple to do from and Eclipse 3. These applications have to be shared on SVN, so maybe zc.buildout might be a better choice in this sense.
Thanks a lot, Andrea
On 10/25/2011 11:05 AM, Andrea Crotti wrote:
After reading everything was possible to read on setuptools and friends, I'm a bit confused...
I have the following situation: suppose that there is a framework composed by many many eggs, which doesn't change very often.
Then there are N applications using this framework, where every application is also composed by 2-3 eggs, and everything is run in a "funny" way (using Envisage).
So at the moment every application fiddles around with the global easy_install.pth, which is bad and needs to be changed, and I have to figure out how.
My idea would be to
- have a global pypi-server running on localhost
- every application is in a virtualenv (or buildout environment) and uses this local pypi-server
I'm doing a few experiments, but already I'm not able to make my virtual env to fetch from my local pypi-server, what is the magic setting for that?
The other future problems that I still see now are:
- I have to be able to develop also the eggs from the framework
- I have to make everything very simple to do from and Eclipse
- These applications have to be shared on SVN, so maybe zc.buildout might be a better choice in this sense.
Thanks a lot, Andrea
This link answers http://jacobian.org/writing/when-pypi-goes-down/ to one of my questions, so now I know at least how to configure buildout and globally easy_install...
On Tue, Oct 25, 2011 at 6:05 AM, Andrea Crotti andrea.crotti.0@gmail.comwrote:
I have the following situation: suppose that there is a framework composed by many many eggs, which doesn't change very often.
Then there are N applications using this framework, where every application is also composed by 2-3 eggs, and everything is run in a "funny" way (using Envisage).
So at the moment every application fiddles around with the global easy_install.pth, which is bad and needs to be changed, and I have to figure out how.
If you use the "-m" option to easy_install, the easy-install.pth won't be changed (or even created, if it doesn't exist). Instead, the application scripts will simply add the needed eggs to their path at runtime.
This is the most flexible approach; the only downside to it is if you just start a Python interpreter, none of the eggs will be on sys.path unless you explicitly pkg_resources.require() them. But for stable app deployment, or even app development (using "setup.py develop" or "setup.py test") it works rather well.
The other future problems that I still see now are:
- I have to be able to develop also the eggs from the framework
Use "setup.py develop -m" to install a project in development mode, without altering easy-install.pth. Any changes made in the source then take effect immediately in the target environment.
- I have to make everything very simple to do from and Eclipse
The "develop" command only needs to be re-run if you change your setup.py options or your dependencies, so it's not hard to do.
- These applications have to be shared on SVN, so maybe zc.buildout
might be a better choice in this sense.
To me buildout sounds like overkill - I have a whole bunch of projects I develop that are installed with "setup.py develop" on my development machine. Any changes I make to the source are immediately seen by any other projects sharing the same targeted installation directory.
Thanks a lot, Andrea ______________________________**_________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/**mailman/listinfo/distutils-sighttp://mail.python.org/mailman/listinfo/distutils-sig
On 10/25/2011 09:59 PM, PJ Eby wrote:
If you use the "-m" option to easy_install, the easy-install.pth won't be changed (or even created, if it doesn't exist). Instead, the application scripts will simply add the needed eggs to their path at runtime.
This is the most flexible approach; the only downside to it is if you just start a Python interpreter, none of the eggs will be on sys.path unless you explicitly pkg_resources.require() them. But for stable app deployment, or even app development (using "setup.py develop" or "setup.py test") it works rather well.
Thanks a lot, that might be a nice solution. Reading the doc I would have never guessed it since "-m" stands for multi-version...
Anyway the other problem is that our applications are actually Envisage applications (which have plugin discovery at run-time) which complicates things a little bit. I'll try to make a trivial example and see if it might work...
However, I still need a pypi-server running on localhost to discover all the eggs I might need to require, is that correct?
To me buildout sounds like overkill - I have a whole bunch of projects I develop that are installed with "setup.py develop" on my development machine. Any changes I make to the source are immediately seen by any other projects sharing the same targeted installation directory.
Yes maybe it's a bit overkill, and involves more complexity, I'll first try the "easy" way...
On Wed, Oct 26, 2011 at 6:21 AM, Andrea Crotti andrea.crotti.0@gmail.comwrote:
On 10/25/2011 09:59 PM, PJ Eby wrote:
If you use the "-m" option to easy_install, the easy-install.pth won't be changed (or even created, if it doesn't exist). Instead, the application scripts will simply add the needed eggs to their path at runtime.
This is the most flexible approach; the only downside to it is if you just start a Python interpreter, none of the eggs will be on sys.path unless you explicitly pkg_resources.require() them. But for stable app deployment, or even app development (using "setup.py develop" or "setup.py test") it works rather well.
Thanks a lot, that might be a nice solution. Reading the doc I would have never guessed it since "-m" stands for multi-version...
Anyway the other problem is that our applications are actually Envisage applications (which have plugin discovery at run-time) which complicates things a little bit. I'll try to make a trivial example and see if it might work...
However, I still need a pypi-server running on localhost to discover all the eggs I might need to require, is that correct?
Why would you need that?
As long as the eggs are present in a sys.path directory, the normal pkg_resources machinery will find them.
If you mean you want to be able to test installation and deployment, or you are continually rebuilding your install directory for some other reason, then once again you don't need a PyPI server - just put all the eggs in a directory somewhere and use the --find-links (-f) option to easy_install to specify that directory name. It will then copy them from there.
On 10/26/2011 03:19 PM, PJ Eby wrote:
Why would you need that?
As long as the eggs are present in a sys.path directory, the normal pkg_resources machinery will find them.
If you mean you want to be able to test installation and deployment, or you are continually rebuilding your install directory for some other reason, then once again you don't need a PyPI server - just put all the eggs in a directory somewhere and use the --find-links (-f) option to easy_install to specify that directory name. It will then copy them from there.
I was convinced that I needed to add every egg to the sys.path to be able to import it, well if I don't actually need the local pypi-server than even better... I was, however, doing some experiments, and not figuring out how it works.
I created an egg_directory where I added the mock library egg, I wanted to see if I was able to find it, and that's what I do.
from pkg_resources import working_set, Environment, Requirement from os.path import abspath
env = Environment(abspath('egg_directory')) env.scan()
r = Requirement.parse('mock') print(working_set.find(r))
It actually finds something but my USER installed version, not the local version in the subdirectory. And from my understanding of the doc it should not even look for in the global sys.path, am I doing something wrong?
Answering to myself, I was using the wrong functions for pkg_resources, and in theory I should use find_distributions. So I tried the following - construct a very minimal egg with a foolproof module - find and add to the working_set all the distributions found
for d in find_distributions('egg_directory'): working_set.add(d)
from foolproof import fool
And in fact in sys.path the foolproof egg is there, but then I can't still import foolproof... Is there anything else I need to do?
Here is what the egg look like.. -rw-r--r-- 47 27-Oct-2011 16:03:16 foolproof.py -rw-r--r-- 291 27-Oct-2011 16:11:44 foolproof.pyc -rw-r--r-- 153 27-Oct-2011 16:11:44 EGG-INFO/SOURCES.txt -rw-r--r-- 1 27-Oct-2011 16:11:44 EGG-INFO/zip-safe -rw-r--r-- 183 27-Oct-2011 16:11:44 EGG-INFO/PKG-INFO -rw-r--r-- 1 27-Oct-2011 16:11:44 EGG-INFO/dependency_links.txt -rw-r--r-- 10 27-Oct-2011 16:11:44 EGG-INFO/top_level.txt
On 10/27/2011 04:16 PM, Andrea Crotti wrote:
Answering to myself, I was using the wrong functions for pkg_resources, and in theory I should use find_distributions. So I tried the following
- construct a very minimal egg with a foolproof module
- find and add to the working_set all the distributions found
for d in find_distributions('egg_directory'): working_set.add(d)
from foolproof import fool
Again answering to myself with this it appears to work, but on the pkg_resources doc for d in find_distributions('egg_directory'): working_set.add(d) d.activate()
or an alternative is to require specifically what I want. Now I need to try from another egg using this egg and see how setup.py tricks work in this sense.
On Thu, Oct 27, 2011 at 11:16 AM, Andrea Crotti andrea.crotti.0@gmail.comwrote:
Answering to myself, I was using the wrong functions for pkg_resources, and in theory I should use find_distributions. So I tried the following
- construct a very minimal egg with a foolproof module
- find and add to the working_set all the distributions found
for d in find_distributions('egg_**directory'): working_set.add(d)
from foolproof import fool
And in fact in sys.path the foolproof egg is there, but then I can't still import foolproof... Is there anything else I need to do?
You're making this way too hard. ;-)
All of the APIs you're using are low-level things, used only for advanced plugin frameworks or installer tools. All you need for a simple script is:
sys.path.insert(0, abspath('egg_directory')) import pkg_resources pkg_resources.require('foolproof') import foolproof
(Note: if you want 'egg_directory' to override the default sys.path, it is important that you *not* import pkg_resources until after you've set up the path.)
Also, even the above is *completely unnecessary* as long as the script you are running is part of a project with its own setup.py.
If your script is listed in a setup.py, and you install it with setup.py develop or using easy_install, then setuptools creates a wrapper for your script that does all of the pkg_resources stuff for you automatically, using the dependences specified in your setup.py. 99% of the time, if you are using pkg_resources for anything but accessing file resources or implementing some sort of plugin system, you are probably doing it wrong. ;-)
So, the only reason I'm even listing the above require() call is just in case you can't run those generated script wrappers easily from inside Eclipse. And, even if that is the case, you should still make sure your setup.py includes those dependencies, so that when you distribute your project the dependencies will still get installed, and a proper wrapper will be generated during installation.
On 10/27/2011 04:34 PM, PJ Eby wrote:
You're making this way too hard. ;-)
All of the APIs you're using are low-level things, used only for advanced plugin frameworks or installer tools. All you need for a simple script is:
sys.path.insert(0, abspath('egg_directory')) import pkg_resources pkg_resources.require('foolproof') import foolproof
Thanks, that's much better right :) So anyway I do have to work with plugins, specifically envisage plugins.
Moreover, our applications are normally composed by more than one egg, and each of them has its own setup_requires requiring a set of the global eggs.
At the moment the run system is a bit complicated and uses pkg_resources.find_plugins...
That said reading the API was still interesting, and for example I found the
working_set.subscribe
In theory I can register a callback function that when something is "found" does something, and I thought immediately that it might be possible to hot-swap some plugins at run-time, is that true doing some tricks with that, is that correct?
def mycallback(dist): if dist == 'name': # reload(dist), if anything like this can be done at all
On Thu, Oct 27, 2011 at 11:58 AM, Andrea Crotti andrea.crotti.0@gmail.comwrote:
Thanks, that's much better right :) So anyway I do have to work with plugins, specifically envisage plugins.
Moreover, our applications are normally composed by more than one egg, and each of them has its own setup_requires requiring a set of the global eggs.
Er, do you mean install_requires? setup_requires is for build-time plugins only.
At the moment the run system is a bit complicated and uses
pkg_resources.find_plugins...
That said reading the API was still interesting, and for example I found the
working_set.subscribe
In theory I can register a callback function that when something is "found" does something, and I thought immediately that it might be possible to hot-swap some plugins at run-time, is that true doing some tricks with that, is that correct?
def mycallback(dist): if dist == 'name': # reload(dist), if anything like this can be done at all
Properly reloading changed code is a lot more complex than that.
On 10/27/2011 06:17 PM, PJ Eby wrote:
Properly reloading changed code is a lot more complex than that.
Ah well it would be nice maybe but it's not so important, and maybe even Envisage can do something like that.
Anyway I was trying a simple example of using "easy_install -m", to see how it works.
I create a minimal library and install it with "-m", so in theory I thought that I can require it and then use it and I do: In [15]: pkg_resources.require('lib') Out[15]: [lib 0.1 (/home/andrea/.local/lib/python2.7/site-packages/lib-0.1-py2.7.egg)]
But then "import lib" or "from lib import module" both fail, so am I missing anything else?
As you said in the other email if I have a setup.py I would not even that, but how do I do in practice?
I tried to do an "easy_install --user -i . foolproof" and foolproof actually has "lib" in install_requires.
Now "lib" is already correctly installed with the "multiversion" option, but then if I try to run the script in foolproof (the installed one) I still can't import "lib".
Anything else I should do?
On 10/31/2011 11:49 AM, Andrea Crotti wrote:
On 10/27/2011 06:17 PM, PJ Eby wrote:
Properly reloading changed code is a lot more complex than that.
Ah well it would be nice maybe but it's not so important, and maybe even Envisage can do something like that.
Anyway I was trying a simple example of using "easy_install -m", to see how it works.
I create a minimal library and install it with "-m", so in theory I thought that I can require it and then use it and I do: In [15]: pkg_resources.require('lib') Out[15]: [lib 0.1 (/home/andrea/.local/lib/python2.7/site-packages/lib-0.1-py2.7.egg)]
But then "import lib" or "from lib import module" both fail, so am I missing anything else?
As you said in the other email if I have a setup.py I would not even that, but how do I do in practice?
I tried to do an "easy_install --user -i . foolproof" and foolproof actually has "lib" in install_requires.
Now "lib" is already correctly installed with the "multiversion" option, but then if I try to run the script in foolproof (the installed one) I still can't import "lib".
Anything else I should do?
Ok after som change (not sure what) I can actually import "lib" now from ipython or from a script. But importing it from the script which was built in the egg which actually has "lib" in install_requires still doesn't work...
How can it be? It doesn't add any pkg_resources trick to find the additional library, and in plus forcing the require doesn't work either..
On 10/25/2011 09:59 PM, PJ Eby wrote:
If you use the "-m" option to easy_install, the easy-install.pth won't be changed (or even created, if it doesn't exist). Instead, the application scripts will simply add the needed eggs to their path at runtime.
This is the most flexible approach; the only downside to it is if you just start a Python interpreter, none of the eggs will be on sys.path unless you explicitly pkg_resources.require() them. But for stable app deployment, or even app development (using "setup.py develop" or "setup.py test") it works rather well.
Another thing, how should I call a setup.py from python itself? I don't like too much the idea of using subprocess to pass arguments to something which is already in python, but it appears that this is how it's done...
I've also seen doing for example sys.argv = ['develop'] execfile('setup.py')
which also doesn't look very nice.
And last thing, do you have any example of using "-m" and where/how I have to require the needed packages with pkg_resources?
On Wed, Oct 26, 2011 at 9:22 AM, Andrea Crotti andrea.crotti.0@gmail.comwrote:
On 10/25/2011 09:59 PM, PJ Eby wrote:
If you use the "-m" option to easy_install, the easy-install.pth won't be changed (or even created, if it doesn't exist). Instead, the application scripts will simply add the needed eggs to their path at runtime.
This is the most flexible approach; the only downside to it is if you just start a Python interpreter, none of the eggs will be on sys.path unless you explicitly pkg_resources.require() them. But for stable app deployment, or even app development (using "setup.py develop" or "setup.py test") it works rather well.
Another thing, how should I call a setup.py from python itself? I don't like too much the idea of using subprocess to pass arguments to something which is already in python, but it appears that this is how it's done...
I've also seen doing for example sys.argv = ['develop'] execfile('setup.py')
which also doesn't look very nice.
And last thing, do you have any example of using "-m" and where/how I have to require the needed packages with pkg_resources?
See the official setuptools documentation links at http://pypi.python.org/pypi/setuptools#using-setuptools-and-easyinstall
(Note, however, that if your application consists of scripts that you run from a location they were installed to with either setup.py develop or easy_install, then you don't need to require() them; just list them in your setup.py's install_requires.)
participants (2)
-
Andrea Crotti -
PJ Eby