Mailman 3 April 2009 - Distutils-SIG

Re: [Distutils] Distutils changes - end user requirements (Was: Deprecate MANIFEST.in)
by P.J. Eby 09 Apr '09

09 Apr '09

At 03:03 PM 4/9/2009 +0100, Paul Moore wrote: >2009/4/9 Eric Smith <eric(a)trueblade.com>: > > Paul Moore wrote: > >> > >> An > >> egg->bdist_wininst converter would fix this issue. As would everyone > >> standardising on bdist_wininst (which, as per the previous message, > >> appears to be prefectly feasible given that bdist_wininst seems to be > >> a strict superset of egg...) > > > > I don't think this is true. I don't think bdist_wininst supports the > > buildout use case where I want to install multiple versions of multiple > > packages in different (or even the same) directory. This is a critical use > > case for Zope/Plone apps. > >OK, I'm making that statement based on some earlier comments. I've >probably misunderstood them. > >But I thought easy_install could take a bdist_wininst installer and >use it to make an egg? Yes, it can. You could also, in principle, reverse the process. However, the resulting bdist_wininst wouldn't support multiple versions, because the bdist_wininst infrastructure doesn't support that.

1 0

setuptools still supported on Python 2.3?
by Lennart Regebro 09 Apr '09

09 Apr '09

I noticed when installing setuptools on Python 2.3 that easy_install fails because it tries to import subprocess, which is a new module in 2.4. I personally do not need Python 2.3 support any longer, but the docs say it should be supported, which seems to be only partly true. The rest of setuptools seems to work on Python 2.3 still, although I haven't tried it in practice. But the tests run. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

1 0

setup.py --install-requires?
by Sridhar Ratnakumar 09 Apr '09

09 Apr '09

Is it possible to get the value of install_requires for an arbitrary package without having to parse setup.py? I see that --name, --version, --description, --provides, and so on are available as an argument to setup.py, but --install-requires is missing. Why? For example, zc.catalog declares these dependencies in its setup.py install_requires=['ZODB3', 'pytz', 'setuptools', 'zope.catalog', 'zope.container', 'zope.component', 'zope.i18nmessageid', 'zope.index>=3.5.1', 'zope.interface', 'zope.publisher', 'zope.schema', 'zope.security', ], Is it possible to reliably get this list in a custom Python code without having to parse setup.py?

3 4

setuptools 0.7a for Python 3 fails to build on Windows
by Jason R. Coombs 09 Apr '09

09 Apr '09

Thanks to Lennart Regebro for putting together a build of setuptools for Python 3 here: http://regebro.wordpress.com/2009/02/01/setuptools-and-easy_install-for-pyth on-3/ Unfortunately, the package fails to build on Windows. Refer to the blog post (and comments) for details. The problem occurs when get_script_args calls resource_string for the 'cli.exe' launcher. It turns out that in Python 3, the following command yields the same error: >>> open('setuptools/cli.exe', 'rt').read() resource_string ultimately executes the same command, which is why the build fails. The correct thing to do in this particular case is to open the file in binary mode, which returns `bytes` instead of a Unicode `str`. Unfortunately, I'm not sure what negative impact would come from altering pkg_resources.DefaultProvider._get to always read bytes, and that doesn't strike me as the correct solution. So, I'm writing here to ask: what should be done about pkg_resources in Python 3 to support getting a package resource that's binary data? As I see it, there are a few options, - always have pkg_resources providers return bytes. - read the bytes and try converting to str, falling back to bytes on failure. - require a parameter to indicate what type of content is expected. There are probably others. Any suggestions are appreciated. Regards, Jason

2 3

Re: [Distutils] short circuiting module lookups
by P.J. Eby 08 Apr '09

08 Apr '09

At 11:54 PM 4/7/2009 +1200, Noah Gift wrote: >1. In the case of entry points for setuptools, it actually recurses >into EVERY egg directory in your path, not just the egg you requested, >adds them to your sys.path and additionally looks for four files >inside of every egg. On a laptop on local storage, this doesn't >matter, but when thousands of machines hit the same filer, with many >python processes, bad things happen... Install your eggs with --multi-version, and then only the eggs that are required for the running script will be added to sys.path or have their contents opened. (Installing them as zip files rather than directories may also speed this up.)

5 7

Re: [Distutils] Plugins for the MANIFEST file
by P.J. Eby 08 Apr '09

08 Apr '09

At 02:33 PM 4/8/2009 +0200, Lennart Regebro wrote: >On Wed, Apr 8, 2009 at 14:23, Tres Seaver <tseaver(a)palladion.com> wrote: > > I want *less* stuff (ideally nothing) spelled in imperative Python, with > > some common declarative file replacing both the information currently in > > setup.py and MANIFEST.in. I thought we were in agreement that > > non-introspectable metadata was a Bad Thing(TM)? > > > > http://wiki.python.org/moin/Distutils/StaticMetadata > >For me, the important part is that it's not spread around many files. >I would also tend to prefer some INI-type file format than python >code. On the other hand, python code offers benefits such as being >able to dynamically change the data depending on things like module >availability and python version, etc. That's why my preference is to make the static format oriented for machine-readability rather than human read/write ability. It should then also be possible to generate the static format from Python or from other formats, and to define a standard around either setup.py commands or something else to let an installer tool request generation of the static data. Like WSGI, the lingua franca needn't be pretty, just well-defined. Heck, I'd almost be willing to use XML for the file format, since XML would make it easy to tag files with multiple pieces of information, such as "this is generated documentation that should be installed" vs. "this is documentation source *and* should also be installed" (e.g. a README.txt that's also reSTified to HTML). XML also allows the markup vocabulary to be extended, and unrecognized markup to be ignored, so that tagging files as "installable" in the general case could allow a dumber installer to just install them without needing to know if they're docs or data or message catalogs or what. That being said, if there was a better format than XML (for appropriate values of "better"; i.e. YAML and JSON don't count), I'd be happy with that too.

1 0

Plugins for the MANIFEST file
by Tarek Ziadé 08 Apr '09

08 Apr '09

Hello, I'd like to refactor the current Distutils code that builds the MANIFEST file so it's pluggable, like what setuptools introduced. Distutils would provide some default plugins, and a new global display option --manifest I have started a new task in the wiki for that here : http://wiki.python.org/moin/Distutils/ManifestPluginSystem If anyone wants to work on it, or comment, Cheers Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

4 11

Re: [Distutils] Deprecate MANIFEST.in
by P.J. Eby 08 Apr '09

08 Apr '09

At 01:49 AM 4/6/2009 +0200, Tarek Ziadé wrote: >So basically, if you get a source distribution out there and work on >it in your own DVCS, >you are unable to create a distro. Why not? Aren't there setuptools plugins available for the common DVCSes?

15 87

pkg_resource require does not seem to find my egg
by Nicholas Veeser 07 Apr '09

07 Apr '09

I am working on a tool we call dino which uses sqlalchemy 0.5.3 Its an update of a previous version (called dino) which uses sqlalchemy 0.4.4. For reasons I don't have to go into, I would like to have both tools installed on the same host, with almost no changes to the existing tool. Thus both versions of sqlalchemy installed So my solution seemed to be use pkg_resources and egg's: - leave sqlalchemy 0.4.4 in: /usr/lib64/python2.4/site-packages/sqlalchemy - build sqlalchemy 0.5.3 as an Egg and install into: /usr/lib64/python2.4/site-packages/SQLAlchemy-0.5.3-py2.4.egg - in the root package of the new code, specify the correct version from pkg_resources import require; require("SQLAlchemy>=0.5.0") This does not work. It does not seem to find the egg, and only finds the old version, and then (correctly) fails. Trying to read through the pkg_resource.py, I am trying to make sense why this is, and how to get around it. I am having trouble wrapping my head around some of the terms... Am I doing something obviously wrong? Is there a known bug? I have installed setuptools-0.6_rc7 Here is the traceback Traceback (most recent call last): File "/usr/bin/dinoadm", line 8, in ? import dino.cmd.cli File "/usr/lib64/python2.4/site-packages/dino/__init__.py", line 6, in ? require("SQLAlchemy>=0.5.0") File "/usr/lib64/python2.4/site-packages/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib64/python2.4/site-packages/pkg_resources.py", line 528, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (SQLAlchemy 0.4.4 (/usr/lib64/python2.4/site-packages), Requirement.parse('SQLAlchemy>=0.5.0')) Thanx Nicholas

2 2

short circuiting module lookups
by Noah Gift 07 Apr '09

07 Apr '09

I work off of a rather large NFS infrastructure where thousands of machines are constantly doing things, and recently I discovered a few things about both setuptools and standard Python lookup that are causing problems. I use nosetests, and noticed that it can take up to 10 seconds to 60 seconds to execute nosetests, a lot of this has to do with the load on the file system I am on that is shared by thousands of machines, but I was a bit troubled to notice the following behavior with an strace: 1. In the case of entry points for setuptools, it actually recurses into EVERY egg directory in your path, not just the egg you requested, adds them to your sys.path and additionally looks for four files inside of every egg. On a laptop on local storage, this doesn't matter, but when thousands of machines hit the same filer, with many python processes, bad things happen... 2. Python itself, also looks at quite a few locations in the search for modules. It looks like this behavior with eggs and setuptools makes them virtually unusable in large installations. Is there any advice for people that have my situation, in which flexibility, i.e. path lookups are not that important, what is important is the least possible lookups to find a module, even to the best case scenario, in which I can tell a command line tool, or wrapper script to only look at a specific path, and to never look at any other location? I haven't done much research yet, but thought I would field the question before I went down a rat hole... -- Cheers, Noah

1 0