Mailman 3 April 2008 - Distutils-SIG

ReST formatting errors when registering package can invalidate HTML conversion
by Noah Gift April 6, 2008

April 6, 2008

I thought I would post a generally help tip that Ian Bicking helped me with. I couldn't figure out why my ReST formatted long_description field would not get converted to HTML when I registered my egg with the cheeseshop: python setup.py register It turns out I had a minor error in my ReST formatting, I didn't underline a headline with enough length. So, the moral of the story is if you wonder why you cannot get HTML formatted ReST to display when you register your egg, it is possible you have a formatting error. It was tricky for me to discover, because I did not get any error message when I uploaded to the cheeseshop, and I just missed the minor formatting error when I was viewing the converted HTML document I used to preview what I created. Noah Gift / http://noahgift.com

2 2

--allow-host and local files
by Tarek Ziadé April 6, 2008

April 6, 2008

Hello, while playing with the allow-host options in setuptools, I have noticed that it is restricted to URLs because url_ok() uses urlparse over the regular expressions that are provided This means that it is not possible to allow local folders to be visited since a "file://*" expression for example, will lead to an empty string: >>> import urlparse >>> urlparse.urlparse('file:///tmp/my/local/file.tgz')[1] '' This will make some links blocked and impossible to add as authorized resource: ... Link to file:///tmp/tmpE-LbUpbuildouttests/setuptools/ ***BLOCKED*** by --allow-hosts ... I would like to propose something: I think this would be easy to change by calling URL_SCHEME() over the url in url_ok(), before urlparse is called. If it is not an url we could then consider that the url is "safe" and return immediatly. if you think it is a good idea, i can provide a patch with a test, Regards, Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

1 1

setuptools bug report: _get_unpatched() is way too clever (and also returns the wrong module)
by zooko April 5, 2008

April 5, 2008

Bug report #1: _get_unpatched() returns a module from the system site-packages instead of the module from the PYTHONPATH which ought to have precedence, leading to setuptools-0.6c8 invoking setuptools-0.6a9 in the following stack trace: PYTHONPATH="/home/buildslave/slave-tahoe/dapper/build/support/lib/ python2.4/site-packages" python ./setup.py develop --prefix="/home/ buildslave/slave-tahoe/dapper/build/support" ... Running Nevow-0.9.18/setup.py -q bdist_egg --dist-dir /tmp/ easy_install-rk83KQ/Nevow-0.9.18/egg-dist-tmp-nJ8GSy Traceback (most recent call last): ... File "/home/buildslave/slave-tahoe/dapper/build/ setuptools-0.6c8.egg/setuptools/command/develop.py", line 102, in install_for_development File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/ easy_install.py", line 518, in process_distribution Full details: https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/1427/ steps/compile/logs/stdio Bug report #2: _get_unpatched() is way too clever for me to spend time trying to understand and debug right now: def _get_unpatched(cls): """Protect against re-patching the distutils if reloaded Also ensures that no other distutils extension monkeypatched the distutils first. """ What the heck? I don't want to think about that, so instead we're delisting Ubuntu Dapper as a supported platform, since it is the only one of our supported platforms on which we have this problem. Generalized complaint: There are a lot of things that are too clever in setuptools/ easy_install/eggs. Cleverness makes it work in more situations, which is good, but it also means that it is harder to know when it will fail or to understand why it failed. A simpler system that satisfied 90% of the use cases while failing in more obvious ways would be appreciated. (Provided, of course, that those 90% of use cases include the ones that I really care about.) See my earlier post which proposes such a simplification: http://mail.python.org/pipermail/python-dev/2008-March/078243.html Bug report #3: setuptools/easy_install/eggs needs a bug tracker. (Also, of course, a buildbot and a rich suite of automated unit tests.) Some of this message may have been inappropriate for distutils-sig, but what else am I going to do with this issue before I delist Dapper and forget all about it? I would be willing to host a trac instance to serve as a wiki and issue tracker for setuptools, and a buildbot. (But I don't want to have anything to do with subversion -- I much prefer darcs.) Regards, Zooko

4 9

Re: [Distutils] Easy_install fails, package somehow depends on itself and won't resolve
by Phillip J. Eby April 4, 2008

April 4, 2008

At 02:52 PM 4/3/2008 -0400, Deron Meranda wrote: >The only thing I can think is that the system I'm installing on has a >slightly different >OS version. But it should be (and is) binary compatible. Setuptools currently uses exact matching on OS strings, so that's almost certainly the problem. If you want to contribute code to do platform matching, check out the pkg_resources docs under "Platform Utilities" and then look at the code for the existing Mac OS version compatibility checking. Be sure to include tests in your patch, and the tests (as well as the version comparison code itself) must be able to run on all platforms. Feel free to ask on the list if you run into any problems/questions.

1 0

[zc.buildout] upgrade with changed explicit recipe version
by Hanno Schlichting April 4, 2008

April 4, 2008

Hi. I ran into a problem with zc.buildout while upgrading a customers buildout and wondered if the problem I encountered is a general one. The buildout in question had one recipe pinned down to an exact version as in: [buildout] parts = plone [plone] recipe = plone.recipe.plone==3.0.2 To upgrade this environment I changed the explicit version to 3.0.6: [buildout] parts = plone [plone] recipe = plone.recipe.plone==3.0.6 When rerunning the buildout script I got an error during uninstall of the plone part. It complained that the plone.recipe.plone recipe version 3.0.2 was not available. From my understanding of this, buildout parses the version requirements first and at the point where it wants to uninstall the plone part, only the newer 3.0.6 version of the recipe is available. The workaround was to change .installed.cfg and remove the version restriction, so any version of the recipe would be accepted to uninstall the part, but I wonder if there is a general problem here or if we are misusing the version restrictions? I have the feeling that buildout should either add the old recipe to the environment to satisfy the uninstall requirement or accept newer versions of a recipe to uninstall older versions. Thoughts? Hanno

2 2

[zc.buildout] buildout unable to parse 'simple' view
by Andreas Jung April 3, 2008

April 3, 2008

We're running a local egg repository based on haufe.eggserver. Buildout won't find an egg called 'haufe.recipe.svnsetup' that is stored on the server. The server is configured properly within the find-links section. And I can see a successful HTTP request to the page in the logs. The underlying template generates the following HTML: <html> <body> .... <a href="http://eggs.haufe-ep.de:21111/eggs/download?package=haufe.recipe.subversion…">haufe.recipe.subversion-0.1.0dev_r942-py2.4.egg</a> <br /> <a href="http://eggs.haufe-ep.de:21111/eggs/download?package=haufe.recipe.subversion…">haufe.recipe.subversion-0.1.1dev_r997-py2.4.egg</a> <br /> <a href="http://eggs.haufe-ep.de:21111/eggs/download?package=haufe.recipe.svnbuild&a…">haufe.recipe.svnbuild-0.1.1dev_r1057-py2.4.egg</a> <br /> ... I wonder why buildout won't parse this HTML properly and find the related distribution?! (Error: Couldn't distribution for 'haufe.recipe.svnsetup'). Andreas

2 1

Easy_install fails, package somehow depends on itself and won't resolve
by Deron Meranda April 3, 2008

April 3, 2008

I'm having an issue with easy_install from setuptools 0.6c8 with egg files containing C extensions. I have created an egg from a package which includes C code. When I try to install it on another system though, the egg itself will install, but then easy_install aborts with an error while trying to resolve it's dependencies (which it shouldn't have any dependencies). The created egg file is named pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg easy_installing it though yields an error: ----- $ easy_install -v -H "" pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Processing pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Copying pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg to /...<PATH>.../lib/python2.5/site-packages Removing pysqlite 2.3.2 from easy-install.pth file Adding pysqlite 2.4.1 to easy-install.pth file Saving /...<PATH>.../lib/python2.5/site-packages/easy-install.pth Installed /...<PATH>.../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg Processing dependencies for pysqlite==2.4.1 Searching for pysqlite==2.4.1 Link to http://pypi.python.org/simple/pysqlite/ ***BLOCKED*** by --allow-hosts Couldn't retrieve index page for 'pysqlite' Scanning index of all packages (this may take a while) Link to http://pypi.python.org/simple/ ***BLOCKED*** by --allow-hosts No local packages or download links found for pysqlite==2.4.1 error: Could not find suitable distribution for Requirement.parse('pysqlite==2.4.1') ----- However I can still use the package just fine, and it seems the same require condition works: >>> import pkg_resources >>> pkg_resources.require('pysqlite==2.4.1') [pysqlite 2.4.1 (/...<PATH>.../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg)] >>> import pysqlite2 >>> pysqlite2.__path__ ['/...<PATH>.../lib/python2.5/site-packages/pysqlite-2.4.1-py2.5-hp-ux-B.11.00-9000-800.egg/pysqlite2'] Everything else works just fine too. The C library gets extracted into ~/.python-eggs and using the package works flawlessly. The only thing I can think is that the system I'm installing on has a slightly different OS version. But it should be (and is) binary compatible. System use to build egg: $ uname -s -r -m HP-UX B.11.00 9000/800 System trying to install on: $ uname -s -r -m HP-UX B.11.11 9000/800 I should also note that the python executable (and its libraries) are identical on each system. Python was actually built on the first system, and copied to the second. Yes I know sqlite is built into python 2.5 so pysqlite isn't needed, but this should work anyway, right? Can anybody help me understand what is going on, and how can I get the egg file to install without an error? Thanks -- Deron Meranda

1 0

FW: [issue2513] 64bit cross compilation on windows
by Mark Hammond April 3, 2008

April 3, 2008

FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited! Thanks, Mark -----Original Message----- From: Mark Hammond [mailto:report@bugs.python.org] Sent: Sunday, 30 March 2008 6:01 PM To: mhammond(a)users.sourceforge.net Subject: [issue2513] 64bit cross compilation on windows New submission from Mark Hammond <mhammond(a)users.sourceforge.net>: I've taken the liberty of adding Trent, Christian and Martin to the nosy list as I know they are actively, if reluctantly interested in this. This patch allows the distutils to cross-compile on Windows. It has been tested on x86 and amd64 platforms, with both platforms successfully able to natively and cross-compile extension modules and create binary distributions. To cross-compile, specify '--plat-name' to the build command (valid values are 'win32', 'win-amd64' and 'win-ia64'). This option name was chosen to be consistent with the bdist_dumb command. I've included the docs I added below (which are also in the patch), but note that as with native compilation using distutils, it's not necessary to set any environment variables or do anything else special with your environment to make this work. The patch also adds a x64 target for the 'bdist_wininst' target, which it creates as distutils/command/wininst-9.0-amd64.exe. This executable is necessary even for bdist_wininst to work natively on x64, but is still included here for simplicity. To assist with testing, I've also added a distutils setup.py script to the PC/example_nt directory. This is capable of creating bdist_wininst executables for both native and cross platforms; 'setup.py build --platname=win-amd64 bdist_wininst' will create an amd64 installer on an x86 machine. The patch has not been tested with a Visual Studio environment without cross-compile tools installed - it will obviously fail, but its not clear how ugly this failure will be. Below is the text I added to docs/distutils/builtdist.rst: Cross-compiling on Windows ===================== Starting with Python 2.6, distutils is capable of cross-compiling between Windows platforms. In practice, this means that with the correct tools installed, you can use a 32bit version of Windows to create 64bit extensions and vice-versa. To build for an alternate platform, specify the :option:`--plat-name` option to the build command. Valid values are currently 'win32', 'win-amd64' and 'win-ia64'. For example, on a 32bit version of Windows, you could execute:: python setup.py build --plat-name=win-amd64 to build a 64bit version of your extension. The Windows Installers also support this option, so the command:: python setup.py build --plat-name=win-amd64 bdist_wininst would create a 64bit installation executable on your 32bit version of Windows. Note that by default, Visual Studio 2008 does not install 64bit compilers or tools. You may need to reexecute the Visual Studio setup process and select these tools. ---------- assignee: mhammond components: Distutils files: windows-cross-compile.patch keywords: 64bit, patch messages: 64743 nosy: Trent.Nelson, ctheune, loewis, mhammond severity: normal status: open title: 64bit cross compilation on windows type: feature request versions: Python 2.6 Added file: http://bugs.python.org/file9900/windows-cross-compile.patch __________________________________ Tracker <report(a)bugs.python.org> <http://bugs.python.org/issue2513> __________________________________

3 7

How we can get rid of eggs for 2.6 and beyond
by Phillip J. Eby April 2, 2008

April 2, 2008

So, after having some time to absorb the Python-Dev threads about setuptools, bootstrap, and all the rest, I think I see an opportunity to let people route around the "damage" of eggs, while still making it possible for the people who want to use easy_install or to put dependencies in their setup.py to get what they want, too. (And without them needing to install eggs, either.) At the same time, we can address the issues that remain around uninstalling packages, system vs. user packages, PYTHONPATH and site.py woes, and really pretty much every other complaint I've heard in the last few days about setuptools stomping on other people's stuff. (Even Paul's Windows issues, hopefully.) Now, you might be asking, "Okay, so why are you telling me about this? Why not just go fix setuptools?" Well, I *can't*. Not without some help from Python-Dev and the Distutils-SIG, to create an updated standard for installed package metadata, by updating PEP 262 ("A Database of Installed Python Packages") to include input from the system packaging folks, support for namespace packages, and support for setuptools-compatible dependency information. What's that got to do with anything? Well, without it, setuptools can't support uninstall or conflict management without using eggs to compartmentalize the installed files. And because it has to use eggs to do *that*, it has to munge .pth files and install its own site.py when installing to PYTHONPATH. All of this ugliness follows directly from the absence of a PEP 262-style installation database. Sure, setuptools could create its own version of this, and I almost did that four years ago. (If you look at the "open issues" part of PEP 262, you'll see my comments from back then.) I decided not to for two reasons: first, the distutils didn't support it yet, so it didn't help for conflict detection and avoidance in the real world at that point. Second, there were no uninstall tools for it, so I'd have had to write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it ain't easy, and I have an aversion to deleting stuff on people's systems without knowing what will break. There's a big difference between them typing 'rm -rf' themselves, and me doing it.) However, if tools exist and are distributed for such a "database", and *everybody* agrees to use it as an officially-blessed standard, then it should be possible for setuptools to co-exist with that framework, and we're all happy campers. In particular, the "installing eggs sucks" camp should be happy, because it'll be possible for me (or anyone else) to write a version of easy_install that doesn't install eggs any more, and setuptools-based packages can go back to having "setup.py install" install things the old way by default. So, to accomplish this, we (for some value of "we") need to: 1. Hash out consensus around what changes or enhancements are needed to PEP 262, to resolve the previously-listed open issues, those that have come up since (namespace packages, dependency specifications, canonical name/version forms), and anything else that comes up. 2. Update or replace the implementation as appropriate, and modify the distutils to support it in Python 2.6 and beyond. And "support it" means, "ensure that 'install' and *all* bdist commands update the database". The bdist_rpm, bdist_wininst, and bdist_msi commands, even bdist_dumb. (This should probably also include the add/remove programs stuff in the Windows case.) 3. Create a document for system packagers referencing the PEP and introducing them to what/why/how of the standard, in case they weren't one of the original participants in creating this. It will probably take some non-trivial work to do all this for Python 2.6, but it's probably possible, if we start now. I don't think it's critical to have an uninstall tool distributed with 2.6, as long as there's a reasonable way to bootstrap its installation later. Questions, comments... volunteers? :)

24 74

Making setuptools play nicer with virtualenv
by Gael Varoquaux April 2, 2008

April 2, 2008

Hi, I have just out-clevered myself using setuptools and virtualenv: * install foo using "python setup.py develop" (foo being ipython). * download some module bar you want to work on in an isolated environment * create this isolated environment using "virtualenv bar" * in the isolated environment "python setup.py develop" the bar module. * still in the isolated environment, try to import bar in a script installed by foo (aka ipython) --> fails because foo uses the system python, and virtualenv wants you to use its own python One very easy solution to make this work is to have the setuptools generated scripts use, under unices, "#!/usr/bin/env python" rather than "#!/usr/bin/python". This seems to me like a good solution, in general, to follow the user's expectations. Is this a change that would be possible? Cheers, Gaël

3 2