development egg and --find-links easy_install option
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. Reading easy_install documentation I recently discovered the - --find-links option, and I'm starting to experiment with it in order to create a tar archive with all the dependencies of a Python package. This is very useful when I have to deploy a package on a shared hosting, where can be a problem to compile extension modules. Unfortunately in my setup I use some packages (the main package being installed and one of its install requirements) in "develop" mode. These packages, moreover, are not available on PyPi. To summarize, I have a package A (to be installed in develop mode), that depends on package B, installed in develop mode. B, then, have additional install requirements, as "normal" packages. A have additional install requirements, as "normal" packages. When I run, from A source directory, the command: python setup.py develop -maxd deps/ an A.egg-link file is correctly copied to the `deps` directory. However, when processing B package, I get: Skipping development or system egg: B 0.1dev Reading http://pypi.python.org/simple/B/ ... Using the `-H none` option, will skip processing of B package and additional dependencies (including direct A dependencies). What is the reason why a development egg is being skipped? What I want is that in `deps` directory should copied all "normal" packages, as eggs, that are install requirements for package A or, indirectly, package B. Is this possible? Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv3/mQACgkQscQJ24LbaUSpbACfcp0gFJhxtlZpjR5D5XQg7q8J FKwAn1ddZhdzQorR+CxGSt30sW+qRKuW =pJ0V -----END PGP SIGNATURE-----
At 05:55 PM 5/22/2010 +0200, Manlio Perillo wrote:
What is the reason why a development egg is being skipped?
Because it can't be copied (currently), and the -a flag in your -maxd means "--always-copy" -- that is, copy the egg, even if it's on sys.path.
What I want is that in `deps` directory should copied all "normal" packages, as eggs, that are install requirements for package A or, indirectly, package B.
Is this possible?
Yes, just build eggs for your local packages and place them somewhere accessible to --find-links. Alternatively, ensure that all needed packages are on sys.path, and use -mxd instead of -maxd. (i.e., allow setuptools to *not* copy things that are already available on sys.path.)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 P.J. Eby ha scritto:
At 05:55 PM 5/22/2010 +0200, Manlio Perillo wrote:
What is the reason why a development egg is being skipped?
Because it can't be copied (currently),
And I usually don't need it to be copied, since on the production server I will install these package by hand, from a Mercurial working copy.
and the -a flag in your -maxd means "--always-copy" -- that is, copy the egg, even if it's on sys.path.
What I want is that in `deps` directory should copied all "normal" packages, as eggs, that are install requirements for package A or, indirectly, package B.
Is this possible?
Yes, just build eggs for your local packages and place them somewhere accessible to --find-links. Alternatively, ensure that all needed packages are on sys.path, and use -mxd instead of -maxd. (i.e., allow setuptools to *not* copy things that are already available on sys.path.)
But the idea was to copy all required packages to a common directory, so that I can create a tar archive from this directory and use it on the production server. And to have this done automatically, without having to build each required package. The needed packages are usually already on sys.path, since they have been installed the first time I executed "python setup.py develop" for my package, on my development machine. The trivial solution is of course to not use "develop" command, and to build a normal egg. I would like that setuptools will just copy the .egg-info file, *making sure* to use a relative path instead of an absolute path. As an example, using virtualenv I have this layout: A/ env/ bin/ include/ lib/ hg/ B/ deps/ setup.py ... `env` is the virtual environment. Inside the `hg` directory I have clones of the packages required by A and that are still under heavy development; these packages are installed using "develop" command. Inside the `deps` directory, I have all the eggs required by A and B packages. I copy all required eggs in this directory, using `-maxd` options, on my development machine. Then I create a tar archive, and, on the production server, I recreate the directory. The virtual environment is recreated each time, I do not use the - --relocatable option. If `-maxd` will make the B.egg-info file relative, then all I need to do is to `python setup.py develop` inside A directory, and B package should be correctly available. Is this possible? Thanks and sorry for the long message Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv4QQYACgkQscQJ24LbaUSJqwCeMEuPGWJHyUyVsSFuevw1V/n1 FBUAnA0XcskGAKwTbmN5tqEVLhhMtG3L =G3Dn -----END PGP SIGNATURE-----
At 10:39 PM 5/22/2010 +0200, Manlio Perillo wrote:
The trivial solution is of course to not use "develop" command, and to build a normal egg.
Right. The slightly-less-trivial version is to make sure your source is in subversion, and add svn: links to your --find-links.
If `-maxd` will make the B.egg-info file relative, then all I need to do is to `python setup.py develop` inside A directory, and B package should be correctly available.
Is this possible?
The -a in -maxd means that you must have either a source distribution (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do .egg-info at the moment (although when it grows PEP 376 support in 0.7 it probably will). The easiest way to do this would be to add something like: svn://myprivateserver/svndir/projectname/trunk#egg=projectname-dev To your --find-links, and set your requirements to "projectname>=NN.NN,==dev", where NN.NN is the version you actually need, and the ==dev part allows it to pull the version from SVN and install it. (Of course, if you don't have or use svn, this doesn't work; at the moment there isn't support for adding new URL protocols to easy_install, although it's on my list for 0.7.) If this way doesn't work, you'll need to either use sdist, or make sure you do an egg install of your local packages before attempting to build a release. Btw, there's still one MORE way to do this: easy_install -maxd targetdir sourcedir1 sourcedir2 ... Where sourcedir1, sourcedir2, etc. are local paths to project directories containing setup.py files. They'll all be built eggs and put in the targetdir. (If you do this, however, you should list the dependencies *first*, rather than last, otherwise it will not be able to resolve them, if you are still only using 'develop' installs there.)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 P.J. Eby ha scritto:
At 10:39 PM 5/22/2010 +0200, Manlio Perillo wrote:
The trivial solution is of course to not use "develop" command, and to build a normal egg.
Right. The slightly-less-trivial version is to make sure your source is in subversion, and add svn: links to your --find-links.
Unfortunately I no more use Subversion. Do you plan to add full support to other VCS, or should I switch to a setuptools fork?
If `-maxd` will make the B.egg-info file relative, then all I need to do is to `python setup.py develop` inside A directory, and B package should be correctly available.
Is this possible?
The -a in -maxd means that you must have either a source distribution (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do .egg-info at the moment (although when it grows PEP 376 support in 0.7 it probably will).
Is it .egg-info or .egg-link ? If it is an .egg-link, then this could be copied as with normal eggs, of course promising (to setuptools) that the path is linked to a valid directory. I tried right now, and it is possible to specify a relative path in the .egg-link file.
[...]
Btw, there's still one MORE way to do this:
easy_install -maxd targetdir sourcedir1 sourcedir2 ...
Where sourcedir1, sourcedir2, etc. are local paths to project directories containing setup.py files. They'll all be built eggs and put in the targetdir.
(If you do this, however, you should list the dependencies *first*, rather than last, otherwise it will not be able to resolve them, if you are still only using 'develop' installs there.)
What do you mean by "first"? Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv4XaQACgkQscQJ24LbaUSkvgCgiAoOIl7oVAJhMOqsndAQ2wk1 Lj4AnjQz47/UT80gGYtudjSbUd6a82ZY =sB+U -----END PGP SIGNATURE-----
At 12:41 AM 5/23/2010 +0200, Manlio Perillo wrote:
P.J. Eby ha scritto:
At 10:39 PM 5/22/2010 +0200, Manlio Perillo wrote:
The trivial solution is of course to not use "develop" command, and to build a normal egg.
Right. The slightly-less-trivial version is to make sure your source is in subversion, and add svn: links to your --find-links.
Unfortunately I no more use Subversion. Do you plan to add full support to other VCS, or should I switch to a setuptools fork?
As I mentioned, support for URL prefix plugins will be added in 0.7. (Not necessarily in the first alpha though.)
If `-maxd` will make the B.egg-info file relative, then all I need to do is to `python setup.py develop` inside A directory, and B package should be correctly available.
Is this possible?
The -a in -maxd means that you must have either a source distribution (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do .egg-info at the moment (although when it grows PEP 376 support in 0.7 it probably will).
Is it .egg-info or .egg-link ?
First off, there really isn't any sane meaning of 'develop' in the context of a -maxd tarball. A -maxd tarball needs .egg files or directories, or else you should just use pip to build a bundle or whatever it's called in pip.
What do you mean by "first"?
I mean, "easy_install -maxd targetdir dependencysource dependersource" - the dependency source directories are to be listed before the source directories that depend on them. All things considered, this is probably the easiest way for you to build a tarball-deployable directory from local source packages that are not indexed on PyPI or pre-built. (I don't know whether pip can handle this use case; i.e., I don't know if it will take source directories as arguments.)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 P.J. Eby ha scritto:
[...]
The -a in -maxd means that you must have either a source distribution (e.g. an sdist .tgz, svn: link, etc.) or an .egg. It cannot do .egg-info at the moment (although when it grows PEP 376 support in 0.7 it probably will).
Is it .egg-info or .egg-link ?
First off, there really isn't any sane meaning of 'develop' in the context of a -maxd tarball. A -maxd tarball needs .egg files or directories, or else you should just use pip to build a bundle or whatever it's called in pip.
pip bundles only contain source distributions. So, it only solve the "I don't have network access" problem, but not the "I can not compile extension modules" problem. Looking at setuptools source code, I think I have solved the problem. The easy_install command when calling the `package_index.fetch_distribution` function, will set the `develop_ok` parameter to False, when `always_copy` option is True. I patched setuptools to always set the parameter to True, and now doing: python setup.py develop -H none -amxd deps do what I want. System and development eggs will no more be skipped, thus causing a DistutilsError exception to be raised; instead the currently installed version will be used. Since I always use `virtualenv --no-site-packages`, all non standard modules will be copied. I will probably implement a custom "bundle" distutils command, that will do this, creating an "eggs bundle" archive (using the sdist --formats option). Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkv5OrEACgkQscQJ24LbaUThiwCfVp3qJY1xu5JedxHVzFoGScek lAAAn3zdVwYiyzp8v7xTg57GjR3DX7aB =ctTd -----END PGP SIGNATURE-----
participants (2)
-
Manlio Perillo -
P.J. Eby