Mailman 3 June 2006 - Distutils-SIG

easy_install download logic?
by Michał Kwiatkowski 21 Jun '06

21 Jun '06

Hi, I'm wondering how exactly easy_install looks up packages on PyPI. Right now I'm looking at two specific packages: moofx and SQLObject. Both have "Home page" link and both have eggs distributions listen on PyPI. The only difference is that moofx has also a "Download URL", which seems to be bad. When I do "easy_install SQLObject" it looks at PyPI, then link from "Home page" field and fall backs to egg: ===== $ sudo easy_install SQLObject==0.7.0 Searching for SQLObject==0.7.0 Reading http://www.python.org/pypi/SQLObject/ Reading http://sqlobject.org Best match: SQLObject 0.7.0 Downloading http://cheeseshop.python.org/packages/2.3/S/SQLObject/SQLObject-0.7.0-py2.3… ===== But this doesn't happen for moofx. It looks at PyPI, then at "Home page", then at "Download URL" and fails, ignoring eggs that are there: ===== $ sudo easy_install moofx Searching for moofx Reading http://www.python.org/pypi/moofx/ Reading http://moofx.mad4milk.net/ Reading http://www.turbogears.org/cogbin/ No local packages or download links found for moofx error: Could not find suitable distribution for Requirement.parse('moofx') ===== Having broken download URL is a bad thing, but shouldn't easy_install use eggs when they're available? And why search "Homepage" and "Download URL" links when eggs are already on PyPI? Thanks for any help, mk -- . o . >> http://joker.linuxstuff.pl << . . o It's easier to get forgiveness for being wrong o o o than forgiveness for being right.

1 0

Re: [Distutils] easy_install fails for Amara XML Toolkit
by Phillip J. Eby 20 Jun '06

20 Jun '06

At 10:58 PM 6/20/2006 +0200, =?ISO-8859-2?Q?Micha=B3_Kwiatkowski?= wrote: >easy_install from setuptools 0.6b3 fails when trying to download Amara >XML Toolkit from PyPI. Full backtrace in attachement. It appears that ftp:// URLs opened by urllib2 don't get any content-type information. I'll fix this in the next release. In the meantime, if you change this line: File "/usr/lib/python2.3/site-packages/setuptools-0.6b3-py2.3.egg/setuptools/package_index.py", line 165, in process_url if 'html' not in f.headers['content-type'].lower(): to read instead: if 'html' not in f.headers.get('content-type', '').lower(): that should get you out of trouble till then. Thanks.

1 0

easy_install fails for Amara XML Toolkit
by Michał Kwiatkowski 20 Jun '06

20 Jun '06

Hi, easy_install from setuptools 0.6b3 fails when trying to download Amara XML Toolkit from PyPI. Full backtrace in attachement. mk -- . o . >> http://joker.linuxstuff.pl << . . o It's easier to get forgiveness for being wrong o o o than forgiveness for being right.

1 0

[Announce] stdeb - Python to Debian source package conversion utility
by Andrew Straw 19 Jun '06

19 Jun '06

I would like to announce the initial public release of stdeb, which may be found at http://stdeb.python-hosting.com/ . The rest of this email is copied directly from that web-page. stdeb - Python to Debian source package conversion utility ========================================================== stdeb_ ("setuptools debian") produces Debian source packages from Python packages via a new distutils command, ``sdist_dsc``, which produces a Debian source package of a Python package. Automatic defaults are provided for the Debian package, but many aspects of the resulting package can be customized via a configuration file. .. _stdeb: http://stdeb.python-hosting.com/ News ---- 2006-06-19: Version 0.1 Released. See the `download page`_. Invocation ---------- All methods eventually result in a call to the ``sdist_dsc`` distutils command. You may prefer to do so directly:: python -c "import stdeb; execfile('setup.py')" sdist_dsc Alternatively, two scripts are provided:: stdeb_run_setup [options] # calls "python setup.py sdist_dsc [options]" py2dsc [options] mypackage-0.1.tar.gz # uses pre-built Python source package In all cases, a Debian source package is produced from unmodified Python packages. The following files are produced: * ``packagename_versionname.orig.tar.gz`` * ``packagename_versionname-debianversion.dsc`` * ``packagename_versionname-debianversion.diff.gz`` These can then be compiled into binary packages using the standard Debian machinery (e.g. dpkg-buildpackage). Download -------- Files are available at the `download page`_. .. _download page: http://stdeb.python-hosting.com/wiki/Download The subversion repository is available at https://svn.stdeb.python-hosting.com/trunk Background ---------- For the average Python package, its source distribution (python_package.tar.gz created with ``python setup.py sdist``) contains nearly everything necessary to make a Debian source package. This near-equivalence encouraged me to write this little distutils extension, which executes the setup.py file to extract relevant information. This process is made significantly easier through the use of setuptools_. .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools setuptools is used because of the opportunities it provides, although many of these features are currently un(der)-utilized. For example, setuptools could make the job of "Debianizing" python console and gui scripts much easier. I wrote this initially to Debianize several Python packages of my own, but I have the feeling it could be generally useful. It appears similar, at least in theory, to easydeb_ and `Logilab's Devtools`_. .. _easydeb: http://easy-deb.sourceforge.net/ .. _Logilab's DevTools: http://www.logilab.org/projects/devtools Prerequisites ------------- * Python_ 2.3 or greater * setuptools_ * subprocess.py_ (included with Python 2.4, backwards compatible with Python 2.3) .. _Python: http://www.python.org/ .. _subprocess.py: http://svn.python.org/view/python/trunk/Lib/subprocess.py?rev=46651&view=log Customizing the produced Debian source package ---------------------------------------------- stdeb will attempt to provide reasonable defaults, but these are only guesses. To customize the Debian source package produced, you may write config files of the format understood by ConfigParser_. When building each package, stdeb looks for the existance of a ``stdeb.cfg`` file in the ``.egg-info`` directory. You may specify an additional config file with the command-line option --extra-cfg-file. .. _ConfigParser: http://docs.python.org/lib/module-ConfigParser.html Here's an example .cfg file which builds several packages:: [DEFAULT] Debian-Version: 0ads1 [setuptools] Source: python-setuptools [numpy] Source: python-numpy Upstream-Version-Prefix: 0.9.8+ Build-Depends: python-dev, refblas3-dev, lapack3-dev Build-Conflicts: atlas3-base, atlas3-base-dev [matplotlib] # matplotlib doesn't incorporate its SVN version number into sdist-built tarballs. # Therefore, if building the SVN version, substitute the version into the # "Upstream-Version-Suffix" variable and use py2dsc. # (For some reason, "debuild -sa" won't build matplotlib because tk.h isn't found.) Source: python-matplotlib Upstream-Version-Suffix: .dev2500 Build-Depends: python-dev, python-numpy, python-numarray, python-numeric, python-gtk2-dev, tk8.4-dev, libwxgtk2.4-dev Depends: python-gtk2, python-numpy, python-numeric, python-numarray Suggests: gs-gpl [scipy] Source: python-scipy Upstream-Version-Prefix: 0.4.9+ Build-Depends: python-numpy Depends: python-numpy .. _numpy: http://scipy.org/NumPy Using stdeb on stdeb -------------------- There is a chicken-and-egg problem when trying to make a Debian package of stdeb with stdeb. Here's a recipe to avoid it:: # in the stdeb distribution directory (with setup.py) python setup.py sdist python setup.py build PYTHONPATH="build/lib" python stdeb/py2dsc.py dist/stdeb-VERSION.tar.gz TODO ---- * Make output meet `Debian Python Policy`_ specifications or the `new python policy`_. This will include several things, including: - the ability to make custom changelogs - the ability to patch upstream source - the ability to include project-supplied documentation (including license information) as a -doc package - the ability to include project-supplied examples, tests, and data as a separate package - much more not listed * Support python-central_ and/or python-support. * Create (better) documentation * Log output using standard distutils mechanisms * Allow distribution-specific configuration parameters (e.g. numpy-dapper) .. _debian python policy: http://www.debian.org/doc/packaging-manuals/python-policy/ .. _new python policy: http://wiki.debian.org/DebianPython/NewPolicy .. _python-central: http://python-modules.alioth.debian.org/python-central_howto.txt Call for volunteers ------------------- I don't have a lot of time for this. This project stands a very real chance of being only a shadow of its potential self unless people step up and contribute. There are numerous ways in which people could help. In particular, I'd be interested in finding a co-maintainer or maintainer if the project generates any interest. Secondarily, I would appreciate advice from Debian developers or Ubuntu MOTUs about the arcane details of Python packaging. License ------- MIT-style license. Copyright (c) 2006 stdeb authors. See the LICENSE.txt file provided with the source distribution for full details. Authors ------- Andrew Straw, California Institute of Technology <strawman(a)astraw.com>

1 0

Re: [Distutils] setuptools: mishandles install of non-python scripts
by Phillip J. Eby 15 Jun '06

15 Jun '06

At 12:52 PM 6/12/2006 -0400, Phillip J. Eby wrote: >At 05:22 PM 6/9/2006 -0700, Ned Deily wrote: > >In build_scripts.py in distutils.command, distuitils uses the results of > >first_line_re to decide whether to adjust the python value in the shbang > >line of the script: if no "python" is found, distutils installs the > >script without modification. easy_install.py uses the first_line_re > >test from distutils but seems to unconditionally add a wrapper to the > >script during installation regardless of the script's type. > >Who needs shell scripts when you have Python? ;) > >I can (sort of) fix this, but there are some potentially-significant >limitations. There's another wrinkle, as it turns out. It's not sufficient to inspect the first line for a #!python presence - it's perfectly valid and possible to have .py or .pyw scripts that don't have a #! line, especially on Windows. So, the actual condition to test if the program is a Python script should be to compile it. If it's valid Python syntax, easy_install will assume it's a Python script, rather than a shell, batch, or other kind of script file. But, it's *also* necessary to check whether there is a *non*-Python #! line, if the first line begins with #!. It's quite possible for a shell script to be syntactically valid Python, especially if you have a shell prolog hidden in what looks like a Python docstring. This is a trick that I believe is listed in the Python FAQ, although I don't know if it's used in any currently-distributed scripts. But it should be supported. So the full rules for analyzing scripts should be: * If there's a #! line, use first_line_re to tell if it's Python * If there's a '.py' or '.pyw' extension, assume it's Python * Otherwise, try to compile the script. If it compiles, assume it's Python * Otherwise, assume it's not. There is one possiblity I'm leaving out here, which is that you have some other script interpreter (e.g. .bat or .cmd files on Windows) that just magically happen to be syntactically-valid Python. In the .bat/.cmd case, anything more complex than a series of one-word commands with no arguments would fail to be valid Python, so it seems incredibly unlikely that this would ever happen. Actually, there's one other possibility I'm also leaving out, which is that you have a .bat or .cmd or some other script whose first line uses -x to skip that line. In such a case, lines 2-N of the file are valid Python, but line 1 doesn't start with #!. In theory, what this should do is keep the first line and wrap the rest. However, such scripts aren't supported by the distutils, so I'm not going to add them to setuptools for 0.6.

1 0

setuptools: mishandles install of non-python scripts
by Ned Deily 15 Jun '06

15 Jun '06

easy_install doesn't properly handle non-python scripts in a distribution. The problem shows up when using easy_install to install mercurial: <http://cheeseshop.python.org/pypi/Mercurial/> The mercurial distribution includes two scripts, one a python script, the other a shell script: hg - #!/usr/bin/env python hgmerge - #!/bin/sh Using easy_install, both scripts get installed with a python wrapper script. Execution of hgmerge fails when the wrapper tries to run the shell script under python: $ hgmerge Traceback (most recent call last): File "/usr/local/python/bin/hgmerge", line 5, in ? pkg_resources.run_script('mercurial==dbeaa4369121', 'hgmerge') File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-pac kages/setuptools-0.6b3-py2.4.egg/pkg_resources.py", line 407, in run_script self.require(requires)[0].run_script(script_name, ns) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-pac kages/setuptools-0.6b3-py2.4.egg/pkg_resources.py", line 1084, in run_script execfile(script_filename, namespace, namespace) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-pac kages/mercurial-dbeaa4369121-py2.4-macosx-10.4-fat.egg/EGG-INFO/scripts/h gmerge", line 20 if [ -z "$EDITOR" ]; then ^ SyntaxError: invalid syntax If mercurial is installed using distutils directly; python setup.py install both scripts work properly. In build_scripts.py in distutils.command, distuitils uses the results of first_line_re to decide whether to adjust the python value in the shbang line of the script: if no "python" is found, distutils installs the script without modification. easy_install.py uses the first_line_re test from distutils but seems to unconditionally add a wrapper to the script during installation regardless of the script's type. -- Ned Deily, nad(a)acm.org

2 2

Re: [Distutils] Minor bug in PythonEggs setuptools-0.6b3 and earlier)?!
by Phillip J. Eby 14 Jun '06

14 Jun '06

At 12:11 PM 6/14/2006 +0200, Jens Engel wrote: >Hello, > >PythonEggs seem to have a problem access the docstring of the root package. >Note that there is no problem accessing docstrings of contained modules >and packages with Python's help(). The pydoc module (which implements the help() function) doesn't support zip files when listing the contents of a package. (That is, it's not the docstring that there's a problem with, it's help() trying to list the modules from the package.) This is a Python bug that is fixed in Python 2.5. I plan to release a hotfix package for older versions of Python that fixes the bug. In the meantime, you can work around it by using the --always-unzip option when easy_install-ing a package; the package will then be installed as a directory instead of a zipfile. But be sure you need it, because egg directories slow down startup and importing much more than zipfiles do. Alternatively, you can just grab the Python 2.5 version of the pydoc and pkgutil modules and drop them on your PYTHONPATH ahead of the standard library versions. (The 2.5 version of pydoc uses utilities in pkgutil to access packages' module lists.)

1 0

Minor bug in PythonEggs setuptools-0.6b3 and earlier)?!
by Jens Engel 14 Jun '06

14 Jun '06

Hello, PythonEggs seem to have a problem access the docstring of the root package. Note that there is no problem accessing docstrings of contained modules and packages with Python's help(). The following session gives an example what I mean by using (and installing) the funkload package. SYNDROME: Install PythonEgg and verify that help() does no longer work => SYNDROME shell> cd funkload-1.5.0 shell> python setup.py install ... # -- INSTALLS PythonEgg ... # TEST: To access docstring of egg-root package -> FAILS. shell> cd .. shell> python >>> import funkload >>> help(funkload) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/site-packages/setuptools-0.6b3-py2.4.egg/site.py", line 328, in __call__ File "/usr/lib/python2.4/pydoc.py", line 1647, in __call__ self.help(request) File "/usr/lib/python2.4/pydoc.py", line 1691, in help else: doc(request, 'Help on %s:') File "/usr/lib/python2.4/pydoc.py", line 1475, in doc pager(title % desc + '\n\n' + text.document(object, name)) File "/usr/lib/python2.4/pydoc.py", line 295, in document if inspect.ismodule(object): return self.docmodule(*args) File "/usr/lib/python2.4/pydoc.py", line 1052, in docmodule for file in os.listdir(object.__path__[0]): OSError: [Errno 2] No such file or directory: '/usr/lib/python2.4/site-packages/funkload-1.5.0-py2.4.egg/funkload' Ciao, Jens Engel

1 0

bad files extensions of text digests in list archive
by Jérôme Bouat 12 Jun '06

12 Jun '06

I browsed the list archive in: http://mail.python.org/pipermail/distutils-sig/ Some text links are tagged as *.txt.gz but there are simply plain text.

2 2

unrecognised swig options ?
by Risky Martin 10 Jun '06

10 Jun '06

Hi, I am trying to use disutils's features with 'SWIG'... but hit a rather strange problem. To reproduce it: setup.py ------------------------------------------------------ from distutils.core import setup, Extension e = Extension("_foo", sources = ["foo.i", ], swig_opts = ["-outdir .", ]) setup(name="foo", ext_modules=[e, ]) ------------------------------------------------------- building fails like: > python setup.py build running build running build_ext building '_foo' extension swigging foo.i to foo_wrap.c swig -python -outdir . -o foo_wrap.c foo.i swig error : Unrecognized option -outdir . Use 'swig -help' for available options. error: command 'swig' failed with exit status 1 but a copy/paste of the swig command shows that '-outdir' *is* recognised... > swig -python -outdir . -o foo_wrap.c foo.i Unable to find file 'foo.i'. Are the permitted options for SWIG hardcoded in 'distutils' ? RiMa

1 1

Distutils-SIG June 2006