At 03:03 PM 4/9/2009 +0100, Paul Moore wrote:
>2009/4/9 Eric Smith <eric(a)trueblade.com>:
> > Paul Moore wrote:
> >>
> >> An
> >> egg->bdist_wininst converter would fix this issue. As would everyone
> >> standardising on bdist_wininst (which, as per the previous message,
> >> appears to be prefectly feasible given that bdist_wininst seems to be
> >> a strict superset of egg...)
> >
> > I don't think this is true. I don't think bdist_wininst supports the
> > buildout use case where I want to install multiple versions of multiple
> > packages in different (or even the same) directory. This is a critical use
> > case for Zope/Plone apps.
>
>OK, I'm making that statement based on some earlier comments. I've
>probably misunderstood them.
>
>But I thought easy_install could take a bdist_wininst installer and
>use it to make an egg?
Yes, it can.
You could also, in principle, reverse the process. However, the
resulting bdist_wininst wouldn't support multiple versions, because
the bdist_wininst infrastructure doesn't support that.
I noticed when installing setuptools on Python 2.3 that easy_install
fails because it tries to import subprocess, which is a new module in
2.4. I personally do not need Python 2.3 support any longer, but the
docs say it should be supported, which seems to be only partly true.
The rest of setuptools seems to work on Python 2.3 still, although I
haven't tried it in practice. But the tests run.
--
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
Is it possible to get the value of install_requires for an arbitrary
package without having to parse setup.py?
I see that --name, --version, --description, --provides, and so on are
available as an argument to setup.py, but --install-requires is missing.
Why?
For example, zc.catalog declares these dependencies in its setup.py
install_requires=['ZODB3',
'pytz',
'setuptools',
'zope.catalog',
'zope.container',
'zope.component',
'zope.i18nmessageid',
'zope.index>=3.5.1',
'zope.interface',
'zope.publisher',
'zope.schema',
'zope.security',
],
Is it possible to reliably get this list in a custom Python code without
having to parse setup.py?
Thanks to Lennart Regebro for putting together a build of setuptools for
Python 3 here:
http://regebro.wordpress.com/2009/02/01/setuptools-and-easy_install-for-pyth
on-3/
Unfortunately, the package fails to build on Windows. Refer to the blog
post (and comments) for details.
The problem occurs when get_script_args calls resource_string for the
'cli.exe' launcher.
It turns out that in Python 3, the following command yields the same error:
>>> open('setuptools/cli.exe', 'rt').read()
resource_string ultimately executes the same command, which is why the build
fails.
The correct thing to do in this particular case is to open the file in
binary mode, which returns `bytes` instead of a Unicode `str`.
Unfortunately, I'm not sure what negative impact would come from altering
pkg_resources.DefaultProvider._get to always read bytes, and that doesn't
strike me as the correct solution.
So, I'm writing here to ask: what should be done about pkg_resources in
Python 3 to support getting a package resource that's binary data?
As I see it, there are a few options,
- always have pkg_resources providers return bytes.
- read the bytes and try converting to str, falling back to bytes on
failure.
- require a parameter to indicate what type of content is expected.
There are probably others. Any suggestions are appreciated.
Regards,
Jason
At 11:54 PM 4/7/2009 +1200, Noah Gift wrote:
>1. In the case of entry points for setuptools, it actually recurses
>into EVERY egg directory in your path, not just the egg you requested,
>adds them to your sys.path and additionally looks for four files
>inside of every egg. On a laptop on local storage, this doesn't
>matter, but when thousands of machines hit the same filer, with many
>python processes, bad things happen...
Install your eggs with --multi-version, and then only the eggs that
are required for the running script will be added to sys.path or have
their contents opened. (Installing them as zip files rather than
directories may also speed this up.)
At 02:33 PM 4/8/2009 +0200, Lennart Regebro wrote:
>On Wed, Apr 8, 2009 at 14:23, Tres Seaver <tseaver(a)palladion.com> wrote:
> > I want *less* stuff (ideally nothing) spelled in imperative Python, with
> > some common declarative file replacing both the information currently in
> > setup.py and MANIFEST.in. I thought we were in agreement that
> > non-introspectable metadata was a Bad Thing(TM)?
> >
> > http://wiki.python.org/moin/Distutils/StaticMetadata
>
>For me, the important part is that it's not spread around many files.
>I would also tend to prefer some INI-type file format than python
>code. On the other hand, python code offers benefits such as being
>able to dynamically change the data depending on things like module
>availability and python version, etc.
That's why my preference is to make the static format oriented for
machine-readability rather than human read/write ability. It should
then also be possible to generate the static format from Python or
from other formats, and to define a standard around either setup.py
commands or something else to let an installer tool request
generation of the static data.
Like WSGI, the lingua franca needn't be pretty, just
well-defined. Heck, I'd almost be willing to use XML for the file
format, since XML would make it easy to tag files with multiple
pieces of information, such as "this is generated documentation that
should be installed" vs. "this is documentation source *and* should
also be installed" (e.g. a README.txt that's also reSTified to
HTML). XML also allows the markup vocabulary to be extended, and
unrecognized markup to be ignored, so that tagging files as
"installable" in the general case could allow a dumber installer to
just install them without needing to know if they're docs or data or
message catalogs or what.
That being said, if there was a better format than XML (for
appropriate values of "better"; i.e. YAML and JSON don't count), I'd
be happy with that too.
At 01:49 AM 4/6/2009 +0200, Tarek Ziadé wrote:
>So basically, if you get a source distribution out there and work on
>it in your own DVCS,
>you are unable to create a distro.
Why not? Aren't there setuptools plugins available for the common DVCSes?
I am working on a tool we call dino which uses sqlalchemy 0.5.3
Its an update of a previous version (called dino) which uses sqlalchemy
0.4.4.
For reasons I don't have to go into, I would like to have both tools
installed on the same host,
with almost no changes to the existing tool. Thus both versions of
sqlalchemy installed
So my solution seemed to be use pkg_resources and egg's:
- leave sqlalchemy 0.4.4 in:
/usr/lib64/python2.4/site-packages/sqlalchemy
- build sqlalchemy 0.5.3 as an Egg and install into:
/usr/lib64/python2.4/site-packages/SQLAlchemy-0.5.3-py2.4.egg
- in the root package of the new code, specify the correct version
from pkg_resources import require; require("SQLAlchemy>=0.5.0")
This does not work.
It does not seem to find the egg, and only finds the old version, and
then (correctly) fails.
Trying to read through the pkg_resource.py, I am trying to make sense
why this is, and
how to get around it. I am having trouble wrapping my head around some
of the terms...
Am I doing something obviously wrong?
Is there a known bug?
I have installed setuptools-0.6_rc7
Here is the traceback
Traceback (most recent call last):
File "/usr/bin/dinoadm", line 8, in ?
import dino.cmd.cli
File "/usr/lib64/python2.4/site-packages/dino/__init__.py", line 6, in ?
require("SQLAlchemy>=0.5.0")
File "/usr/lib64/python2.4/site-packages/pkg_resources.py", line 626,
in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib64/python2.4/site-packages/pkg_resources.py", line 528,
in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (SQLAlchemy 0.4.4
(/usr/lib64/python2.4/site-packages),
Requirement.parse('SQLAlchemy>=0.5.0'))
Thanx
Nicholas
I work off of a rather large NFS infrastructure where thousands of
machines are constantly doing things, and recently I discovered a few
things about both setuptools and standard Python lookup that are
causing problems.
I use nosetests, and noticed that it can take up to 10 seconds to 60
seconds to execute nosetests, a lot of this has to do with the load on
the file system I am on that is shared by thousands of machines, but I
was a bit troubled to notice the following behavior with an strace:
1. In the case of entry points for setuptools, it actually recurses
into EVERY egg directory in your path, not just the egg you requested,
adds them to your sys.path and additionally looks for four files
inside of every egg. On a laptop on local storage, this doesn't
matter, but when thousands of machines hit the same filer, with many
python processes, bad things happen...
2. Python itself, also looks at quite a few locations in the search
for modules.
It looks like this behavior with eggs and setuptools makes them
virtually unusable in large installations. Is there any advice for
people that have my situation, in which flexibility, i.e. path lookups
are not that important, what is important is the least possible
lookups to find a module, even to the best case scenario, in which I
can tell a command line tool, or wrapper script to only look at a
specific path, and to never look at any other location?
I haven't done much research yet, but thought I would field the
question before I went down a rat hole...
--
Cheers,
Noah
