Mailman 3 June 2012 - Distutils-SIG

building an application made from many internal packages
by Chris Jerdonek June 13, 2012

June 13, 2012

Hi, is there a recommended way of building and deploying a private Python application from many loosely coupled internal packages or modules as opposed to a single large package? Should this be done with a private PyPI server with each module/package having its own setup.py, or is zc.buildout (or some other tool) the more appropriate solution here? Thanks, --Chris

2 1

RFC: package multiple platforms into a single binary package
by Daniel Holth June 13, 2012

June 13, 2012

I've been trying to work out a filename convention and specification for packaging multiple platforms worth of Python into a single archive. The feature would reduce the number of mostly-redundant archives required for a package that is mostly Python but has a small extension module, or express that a package contains fat binaries a-la OS X. Now that Python includes the __pycache__ directory and ABI tags for extension modules, it will often be possible to copy installations of the same package compiled for different Python ABIs on top of each other without clobbering any files. For example, a directory could include .pyc files for Python 3.2 and 3.3, or extension modules for win32 and Linux at the same time. When copied into the same location, the only files clobbered by the second architecture should be the identical .py, .dist-info/METADATA and so forth, while the second platform's shared libraries will have a unique name. The proposed filename is {distribution}-{version}-{python tag}-{platform tag}.{archive extension} distribution: Distribution name, e.g. ‘django’, ‘pyramid’ version: PEP 386-compliant version, e.g. 1.0 Python implementation tag Platform tag or 'noarch' For example, package-1.0-py27-noarch.rar is compatible with Python 2.7 (any Python 2.7 implementation) on any CPU architecture. The Python implementation is abbreviated to save space. Each implementation has a two-letter code: py: Generic Python cp: CPython ip: IronPython pp: PyPy jy: Jython concatenated with the ABI tag: SOABI.split(‘-’)[-1] “33m”, or, for “pure Python” packages, py_version_nodot “27”. The platform tag is distutils.util.get_platform() with all periods and hyphens replaced with underscore, or the string ‘noarch’. The {python tag} field and {platform tag} field are actually .-delimited sets of tags. The filename indicates compatibility with the cartesian product of the two sets. For example, you would be able to combine a win32 and linux_x86_64 archive into package-1.0-py33-linux_x86_64.win32.tar.bz2 The part I'm unsure about is that 'noarch' and 'py27' don't quite fit with 'cp33dmu' (which says something about both the compiled code and the version of Python required by the .py files). Thanks, Daniel Holth

1 0

[issue138] setuptools.extension incompatible w/ Cython (patch included)
by JimC June 12, 2012

June 12, 2012

New submission from JimC <jim(a)carroll.com>: We discovered a minor problem with setuptools.extension.py that prevents being able to incrementally build extenion modules when working with Cython (meaning, edits to source do NOT result in a call to the cython compiler). A patch is attached to this ticket. The problem is the extension.Extension() class has logic that alters the filename the sources[] list if Pyrex is not installed. The code has a simple boolean test if Pyrex is installed, and if not, it remaps the extension from ".pyx" -> ".c". I believe the simple solution is to perform the same logic for Cython (to set a boolean if Cython is available, and to NOT remap the source extension if either Pyrex OR Cython is installed). Example to demonstrate the problem: setup('project', ..., ext_modules = Extension('Prj', ['Prj/pr1.pyx']), ..., cmdclass = {'build_ext': Cython.Distutils.build_ext}, ..., With a setup.py like this, the setuptools.extension.Extension class will re-write the sources[] list BEFORE the Cython.Distutils.build_ext module can get a shot at it. The result is that subsequent edits of 'Prj/pr1.pyx' will cause build_ext to skip the cython compile step. The only work around to force a recompile is to delete the contents of the 'build/' folder. See the suggested patch for our request for modification. Thanks ---------- files: extension.patch messages: 658 nosy: jim(a)carroll.com priority: bug status: unread title: setuptools.extension incompatible w/ Cython (patch included) Added file: http://bugs.python.org/setuptools/file80/extension.patch _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue138> _______________________________________________

1 0

Packaging dependencies
by Keshav Kini June 12, 2012

June 12, 2012

Hi, We're using distribute to package our software, but we'd like to do something different with the "sdist" command. We would like to recursively find all packages necessary to build our package from a base Python install, download these packages, and store them all in a directory inside our source distribution tarball; then, when the tarball is unpacked and setup.py is run, we want "install" to use those cached packages. (This is to support installation on machines that can't or shouldn't access the network.) One solution I found in the distribute docs was to use `easy_install -zmaxd dirname package` on our package to generate and store dependency eggs, and then `easy_install -H none -f dirname` when installing the package. Unfortunately this actually builds packages which have C extensions and stuff like that in them, whereas we would like to only ship the source and build the dependencies when our package is built (to support multiple platforms). Also I guess we'd like `setup.py install` to just work in the directory without the user having to care about the fact that these source packages were shipped inside the tarball they just extracted. So I'm wondering if I can actually do this by importing something from distribute itself and writing some nontrivial code in setup.py. Can someone point me to what would be the best way to do this? I'm guessing I'll need to start by subclassing Environment to create a fake "default environment" which looks like how a bare Python 2.7.3 (say) install would. Then I'd need to somehow modify "sdist" somewhere to make it download the dependency tarballs that would be necessary if you were installing our packages into that Environment, and also need to modify "install" to make it look in the correct directory. Is that about right? Thanks and sorry for the simple question. -Keshav

2 1

bug#14651
by Rob Healey June 10, 2012

June 10, 2012

Greetings: I know that there are many many bugs out there right now, I must sincerely and grateful request that bug#14651's patch would please be accepted and applied before python-3.3.0 freeze happens! The only reasons that I must ask at this point is because without it, it completely stops the work that, me and others, are doing on our project??? -- Sincerely yours, Rob G. Healey

1 0