Mailman 3 August 2009 - Distutils-SIG

Re: [Distutils] Static metadata using setup.cfg
by David Lyon 17 Aug '09

17 Aug '09

On Tue, 18 Aug 2009 06:06:29 +0300, Alex Grönholm <alex.gronholm(a)nextday.fi> wrote: > That scheme does not allow me to say "This dependency is required unless > platform is X". A practical example of this is Beaker, where pycryptopp > is required for cookie encryption, but works without external > dependencies on Jython. Yes, I follow.. That's why I'm so keen on having a pre_setup() and a post_setup() user routine, where the environment and dependencies can be exposed. I can't see any easy way to do conditional logic in a config file. imho it just has to go in code.. and it is as simple as that - haha for example (demonstration code - not runnable): """ from distutils.core import setup def pre_setup(): if sys.platform() == "linux2": self.dependencies.append("pycryptopp") return setup() """ So, one could modify the dependencies dynamically based on the underlying platform. Dependencies is a list of packages within the setup class. So the user can add or remove items before the actual setup() is done. > Also, how do I define that this package has a minimum (or maximum) > required Python version? [Python_versions] minimum_supported=2.3 maximum_supported=2.7 We can't tell a user it won't work - only that it isn't supported. > Or that the distribution is only applicable to a certain platform (say, > win32 or java)? [Platforms] supported1=linux2 supported2=mac supported3=win32,win64 I think we can let the developer specify what *is* supported, and if it doesn't match then give the user the choice to proceed or not. (At their own risk) David

1 0

Make easy_install script updates atomic / allow in multi-user env
by ryles 17 Aug '09

17 Aug '09

In setuptools/command/easy_install.py, easy_install.write_script is the method called to install each of a package's scripts. It contains the following: f = open(target,"w"+mode) f.write(contents) f.close() chmod(target,0755) opening the file with "w" will cause it to truncate, and then there is a race between the open and the write. Further, due to buffering there is even no guarantee that a user will see either an empty or complete version of the script during this time. It should be safer to instead write to a temporary file and then use shutil.move, which, on Unix- like systems at least, will call rename() (i.e. what /bin/mv would do). renaming is atomic and allows the old version to remain in tact (anonymously) if any process had it open for reading, and also ensures that no process will see an incomplete version. The above code can also create problems in a multi-user environment. chmod() is inflexibly giving only the owner write access. The author may have purposely given the script (the original non-wrapper version, at least) group write access so that other developers could install newer versions of the package at a later time. The chmod() call blocks this attempt. More importantly, directory permissions are ignored. That is, even if the script itself is not writable (necessarily true with the current code), its parent directory may purposely be writable so that newer versions can be added. Using shutl.move will address this since it operates on the directory and not the file itself. This is arguable the correct semantics anyway: "install a new version of the script", not "edit the existing script".

1 0

[setuptools] conditionally installing scripts based on extras require
by Sridhar Ratnakumar 17 Aug '09

17 Aug '09

I have a setup.py like this: [...] extras_require = {'foo': 'pkginfo'}, [...] entry_points = {'console_scripts': [ 'bar=myapp.bar:main', 'foo=myapp.foo:main [foo]', ]} [...] When I do a 'python setup.py install', the package 'pkginfo' is *not* installed as expected. However, the script 'foo' *is* installed. This is somewhat ironic .. because `foo' as per the definition depends on the 'foo' feature; yet installing 'myapp' (without 'foo' feature) installs the script 'foo'. 1) What is the rationale behind this behavior? setuptools documentation reads: "[...] if Project A includes a "rst2pdf" script, it might declare it like this, so that the "PDF" requirements are only resolved if the "rst2pdf" script is run". Why install a script if the user installed a distribution without a particular feature that this script depends on? 2) Is there a workaround to tell setuptools *not* to install scripts depending on features not asked for? -srid

1 0

Splitting distribute in several distributions
by Tarek Ziadé 17 Aug '09

17 Aug '09

Hey, I've started to work on splitting Distribute into several distributions for 0.7 Here's the organization I was thinking about: - Distribution Name = "Distribute" - Content: setuptools package renamed to distribute (minus some elements). root modules in setuptools will be moved to the distribute.core package so the root namespace stays empty. - Distribution Name = "DistributeResources" - Content: the part of pkg_resources.py that handles installed distribution querying + renamed to "distribute.resources" - Distribution Name = "DistributeEntryPoints" - Content: the part of pkg_resources.py that handles entry points + renamed to "distribute.entry_points" - Distribution Name = "DistributeInstaller" - Content: the part of pkg_resources.py that handles version comparison + package_index, that gets files at pypi + easy_install.py + easy_install command extracted from setuptools package + renamed into a "distribute.installer" package I am not sure yet how the distribute namespace will be handled, because (maybe ala PEP 382) I am not sure either if "setuptools.sandbox" should get its own distribution. I'd keep it in Distribute Any thoughts ? Cheers Tarek -- Tarek Ziadé | http://ziade.org

13 36

setup.py develop is not locating install_requires packages in custom site dir
by ryles 17 Aug '09

17 Aug '09

I have a package, 'proj', which specifies a dependency on 'mypkg', version 0.0.2 $ cat setup.py from setuptools import find_packages, setup setup(name = "proj", version = "0.0.1", platforms = ["linux2"], packages = find_packages(), zip_safe = False, install_requires=["mypkg==0.0.2"], entry_points = {"console_scripts": ["proj = proj.main:main"]}, ) I have been able to successfully easy_install an egg created with setup.py bdist_egg, and the 'proj' entry point successfully `requires' mypkg 0.0.2. However, setup.py develop is not able to find mypkg. I am installing it into the user site dir so that it's private during development, rather than in my system wide /custom/site-packages directory: $ python setup.py develop --install-dir ~/.local/lib/python2.6/site- packages --script-dir ~/bin --allow-hosts localhost running develop Checking .pth file support in /home/user/.local/lib/python2.6/site- packages /custom/bin/python -E -c pass TEST PASSED: /home/user/.local/lib/python2.6/site-packages appears to support .pth files running egg_info creating proj.egg-info writing requirements to proj.egg-info/requires.txt writing proj.egg-info/PKG-INFO writing top-level names to proj.egg-info/top_level.txt writing dependency_links to proj.egg-info/dependency_links.txt writing entry points to proj.egg-info/entry_points.txt writing manifest file 'proj.egg-info/SOURCES.txt' reading manifest file 'proj.egg-info/SOURCES.txt' writing manifest file 'proj.egg-info/SOURCES.txt' running build_ext Creating /home/user/.local/lib/python2.6/site-packages/proj.egg-link (link to .) Adding proj 0.0.1 to easy-install.pth file Installing proj script to /home/user/bin Installed /path/to/proj Processing dependencies for proj==0.0.1 Searching for mypkg==0.0.2 Link to http://pypi.python.org/simple/mypkg/ ***BLOCKED*** by --allow- hosts Couldn't find index page for 'myproj' (maybe misspelled?) Scanning index of all packages (this may take a while) Link to http://pypi.python.org/simple/ ***BLOCKED*** by --allow-hosts No local packages or download links found for mypkg==0.0.2 error: Could not find suitable distribution for Requirement.parse ('mypkg==0.0.2') pkg_resources.require() has no trouble finding it, however. $ python Python 2.6.2 (r262:71600, Jul 16 2009, 14:04:28) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from pkg_resources import require >>> require("mypkg==0.0.2") [mypkg 0.0.2 (/custom/site-packages/mypkg-0.0.2-py2.6.egg)] >>> import mypkg >>> The problem is fixed if I explicitly tell setuptools that /custom/site- packages is a site directory with --site-dirs, but it's unclear why this should be necessary since that directory is already in sys.path. Note that / custom/site-packages was created by following the instructions here: http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installat…

1 2

Re: [Distutils] Why is requiring version 0.0.1 when both 0.0.1 and 0.0.2 are available a conflict?
by P.J. Eby 17 Aug '09

17 Aug '09

At 12:45 AM 8/17/2009 -0700, ryles wrote: >Can someone explain what exactly the conflict is (I'm guessing >pkg_resources is unhappy that 0.0.2 is already on sys.path)? You guess correctly. >I'd like for users to be able to explicitly use the older version if >they're not yet comfortable with the new one. And a user just writing >a simple script without wanting setuptools, or even simply trying >something in the interactive interpreter, should be able to access the >latest version without needing to know about pkg_resources. Whichever version you install later will be the default in the interpreter, unless you use --multi-version. Note, however, that a script whose setup.py requires a non-default version of the package will still work. In other words, if you create another project with a setup.py that requires version 0.0.1, then upon installation that project's scripts will use 0.0.1, even though the interpreter's default will remain 0.0.2.

2 1

Why is requiring version 0.0.1 when both 0.0.1 and 0.0.2 are available a conflict?
by ryles 17 Aug '09

17 Aug '09

I have 2 egg versions of 'mypkg' installed in /custom/site-packages, 0.0.1 and 0.0.2 This directory is an alternate site directory as described here: http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installat… I have elected to not use easy_install --multi-version since it's convenient to have easy-install.pth set up so that users can by default import the latest version if they don't explicitly require a version either via setup.py or by using pkg_resources.require directly. However, require() is refusing to resolve version 0.0.1: Python 2.6.2 (r262:71600, Jul 16 2009, 14:04:28) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. py> from pkg_resources import require py> require("mypkg==0.0.1") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 626, in require File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 528, in resolve pkg_resources.VersionConflict: (mypkg 0.0.2 (/custom/site-packages/ mypkgl-0.0.2-py2.6.egg), Requirement.parse('mypkg==0.0.1')) py> Can someone explain what exactly the conflict is (I'm guessing pkg_resources is unhappy that 0.0.2 is already on sys.path)? I'd like for users to be able to explicitly use the older version if they're not yet comfortable with the new one. And a user just writing a simple script without wanting setuptools, or even simply trying something in the interactive interpreter, should be able to access the latest version without needing to know about pkg_resources.

1 0

Why is requiring version 0.0.1 when both 0.0.1 and 0.0.2 are available a conflict?
by ryles 17 Aug '09

17 Aug '09

I have 2 egg versions of 'mypkg' installed in /custom/site-packages, 0.0.1 and 0.0.2 This directory is an alternate site directory as described here: http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installat… I have elected to not use easy_install --multi-version since it's convenient to have easy-install.pth set up so that users can by default import the latest version if they don't explicitly require a version either via setup.py or by using pkg_resources.require directly. However, require() is refusing to resolve version 0.0.1: Python 2.6.2 (r262:71600, Jul 16 2009, 14:04:28) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. py> from pkg_resources import require py> require("mypkg==0.0.1") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 626, in require File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 528, in resolve pkg_resources.VersionConflict: (mypkg 0.0.2 (/custom/site-packages/ mypkgl-0.0.2-py2.6.egg), Requirement.parse('mypkg==0.0.1')) py> Can someone explain what exactly the conflict is (I'm guessing pkg_resources is unhappy that 0.0.2 is already on sys.path)? I'd like for users to be able to explicitly use the older version if they're not yet comfortable with the new one. And a user just writing a simple script without wanting setuptools, or even simply trying something in the interactive interpreter, should be able to access the latest version without needing to know about pkg_resources.

1 0

Low Level API for translating distutils/setuptools metatdata to Debian metadata
by Brian Sutherland 16 Aug '09

16 Aug '09

Hi, I've just published a very small library that does three things so far: * Provides a mapping between python project names and Debian binary/source package names * Converts setuptools versions to Debian versions while maintaining sort order * Can introspect an .egg-info directory to figure out the Debian dependencies for use in the debian/control file. It can also handle/understand extras (I Hope!) It's basically an attempt to find a common ground between all the projects doing automated Python->Debian packaging. I have a feeling everyone is re-implementing this code all the time. The code is here: http://pypi.python.org/pypi/van.pydeb/ And an example of direct use in a packaging situation with complex dependencies is here: http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?vi… -- Brian Sutherland

5 10

Re: [Distutils] request for comments - standardization of python's purelib and platlib
by Jan Matejek 14 Aug '09

14 Aug '09

Dne 13.8.2009 21:22, Brett Cannon napsal(a): > On Thu, Aug 13, 2009 at 11:23, Jan Matejek <jan.matejek(a)novell.com> wrote: >> 1 - the traditional way >> purelib = /usr/lib/pythonX.Y/site-packages >> platlib = /usr/lib(64)/pythonX.Y/site-packages >> > > Why can't pure libraries go into lib64 as well? There is nothing saying that > a pure Python package won't have a setup.py that installs different files > based on whether it is for a 32-bit or 64-bit CPython install. What i'd like to accomplish is to have pure "noarch" package that can be installed unchanged into 32bit or 64bit (or 256bit) system, and the respective python would still find the files. Or, to put it another way, a package that can be installed into a multiarch system and be recognized by pythons of all architectures (assuming they are the same version, of course). If the distutils package installs different pure files for 32bit and 64bit python, then it can't be "noarch", so it doesn't matter if it goes into lib64. Also, such package would break this particular scheme - in the situation where the user installs only 32bit version of such package and tries to run it with 64bit python, it will probably break in some weird way. Last but not least, i'd argue that if a python-only package installs different files for different platforms, it is platform-dependent and therefore not pure ;) >> 2 - the sharedir way >> purelib = /usr/share/python/X.Y >> platlib = /usr/lib(64)/pythonX.Y/site-packages > > > Now are you proposing that packages that have both Python source and > extensions be split based on the type of files, or that only pure Python > packages go to /usr/share/python and any packages that are mixed go into > lib(64)? If you are proposing the latter this is more reasonable as the > former will require using .pth files to get import to search both locations > for files in the same package and that just feels icky to me. The latter. Assume no change to "normal" distutils mechanism, only setting the default paths. (for now anyway) regards m.

1 0