Mailman 3 November 2005 - Distutils-SIG

C++, swig, and distutils
by Vishnu Bob Nov. 18, 2005

Nov. 18, 2005

Hi, I have a two comments about the usability of C++, swig, and distutils. I'm using python 2.4.2. Comment 1: First, build_ext.py:525 tests the list "swig_opts". If the string "-c++" is in the swig_opts list, build_ext.py will build the wrapper with a ".cpp" extension instead of the default ".c" if self.swig_cpp or ('-c++' in self.swig_opts): target_ext = '.cpp' else: target_ext = '.c' However, "swig_cpp" is a command level option, and it's impossible to set via the Extension() arguments. This gets a little weird, because of build_ext.py:550: # Do not override commandline arguments if not self.swig_opts: for o in extension.swig_opts: swig_cmd.append(o) This is below the "-c++" string test described above. So, if you specify "-c++" like this: Extension( 'package.module', ['src/module.i'], swig_opts=['-c++'], language='c++' ) build_ext.py will include "-c++" into the swig command, but the output file will /still/ have a ".c" extension. In order to workaround this without specifying a command line argument or building a setup.cfg file, you have to specify an "options" keyword argument in the call to setup(): options={'build_ext':{'swig_opts':'-c++'}} Comment 2: In spite of workaround described above, build_ext.py:464 compiles the swig generated ".cpp" wrapper with "gcc". Luckily, gcc sees the ".cpp" extension and switches to g++, but that's totally sloppy. I've tried to coax distutils to use its C++ swig strategies with three different variables, and it's still assuming C. thanks, -g

1 0

Mac OS 10.4
by Robin Becker Nov. 18, 2005

Nov. 18, 2005

Can some knowledgeable person tell me if there are any nasty things to watch out for when building extensions for the standard installed python for Mac OS 10.4 (I believe that's the number corresponding to 'Tiger'). I have built my own Python versions in the past and that worked fine. I just wondered if there are any gotchas relating to the mach loader etc etc. -- Robin Becker

2 3

extra_compile/link_args : split string before exec
by Simon Burton Nov. 17, 2005

Nov. 17, 2005

HI, i just found a nasty bug in my distutils: passing in multiple arguments to a single element in the extra_link_args list. Those multiple arguments are not .split() by distutils before exec'ing gcc and so ld was completely ignoring some essential libs. I would suggest either adding a .split() into distutils, or at least documenting this. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com

1 0

Question about usage of pje's setuptools
by Luc Stepniewski Nov. 17, 2005

Nov. 17, 2005

I have made a program that I want to distribute as an egg. The problem is that I need to put some files in directories like /etc/init.d and some configurations in a newly created directory in /etc. I know there is the "entry_points" + "console_scripts" stuff that I can use in my setup.py, but this is only for putting executables in /usr/bin. I know that setuptools is pretty new, and I understand the reasons why the sandbox refuses to create files outside of it, but if so, how do people develop big applications, that require configuration files, init files (at boot I mean), etc.? I did not find any example of big programs using setuptools that need global installation. Thanks for your help, Luc

2 1

Setuptools 0.6a8 update to fix SourceForge problems
by Phillip J. Eby Nov. 17, 2005

Nov. 17, 2005

FYI, SourceForge changed their download pages such that older versions of setuptools can no longer recognize them and follow the links to the "real" download pages. I'm releasing 0.6a8 early in order to address these issues. You can update your setuptools with: easy_install -U setuptools If you have an 'ez_setup' subdirectory in a Subversion-based project, you can use 'svn update ez_setup' to ensure you're requiring the latest version. Unfortunately, there are still outstanding bugs, specifically the "space in python.exe filename" problem, and the issue with creating RPMs from source distributions that use the source-control-driven MANIFEST feature. However, these bugs are also in older versions, so they aren't new to 0.6a8. And 0.6a8 also fixes a lot of previous issues with doing Subversion-based development, so it's a good upgrade to get anyway. Sorry for the inconvenience; the original SourceForge processing code was a user contribution and I didn't really vet it for robustness in the face of SourceForge changes. The new parsing code could still be broken by future SF changes, but it's now less likely.

2 4

Re: [Distutils] MANIFEST destiny :)
by Phillip J. Eby Nov. 17, 2005

Nov. 17, 2005

At 10:40 AM 11/17/2005 +0000, Paul Moore wrote: >This seems to me to be a good point. However, wouldn't a relatively >simple way of addressing it be just to *document* the One Official Way >(ideally, in the Python documentation, but maybe hosted elsewhere as >well for users of current and older versions)? I'm doing this with the setuptools documentation to some extent now, although I'm not quite so prescriptive as I imply we should be in the larger and long term sense. Mainly that's because right now setuptools adoption depends on early adopters among existing package distributors, and it doesn't make sense for me to dictate e.g. their directory layout, as long as it's halfway sane. :) >You could add a footnote somewhere that setuptools is limited to >support of projects following this approach. Actually, I can't. Setuptools needs to be 100% backward-compatible with existing distributions, at least to the point of building eggs for them. Otherwise, I can't get the critical mass of users needed to bootstrap this thing.

1 0

MANIFEST destiny :)
by Phillip J. Eby Nov. 17, 2005

Nov. 17, 2005

I originally added CVS and Subversion support to setuptools in order to get past the pain of the distutils' MANIFEST system. I used to use MANIFEST.in, but it was a royal pain to get right, and I pretty much always forgot to add stuff to it. The most common problem when I shipped a source distribution was that the MANIFEST was screwed up, such that a CVS checkout worked fine but a source distribution would break. Ugh. So, the CVS/Subversion support for setuptools automatically makes your MANIFEST include anything under revision control, whether you have a MANIFEST.in or not. If you don't have a MANIFEST.in, the MANIFEST is built every time you run an sdist, so it's always up-to-date. Ah, bliss! But all is not happy in MANIFESTville. It turns out that the bdist_rpm command expects to build an sdist, for reasons impenetrable to me. If you build an RPM from a source checkout, everything is fine because setuptools can auto-discover your files and build the MANIFEST for the new sdist. But if you build an RPM *from* an sdist, it's a no-go. In addition, many folks have been asking for this autodetection to cover package data files as well. Why, they reasonably ask, must I specify each and every file to be included in a package, when the system already knows what files I have in revision control, or which is covered by my MANIFEST.in? The reason I've been avoiding adding this feature, however, is because of the first issue; when you make an sdist, you lose that additional metadata, so it would become impossible to build *any* binary from an sdist, not just RPMs. Until recently, that issue seemed insurmountable. So today, after looking over the issue a bit, I think I have a plan for dealing with MANIFEST: * Change the MANIFEST format to be platform-independent (currently it contains OS-specific path separators) * Always, always, always build MANIFEST, and always include both the MANIFEST file and MANIFEST.in (if present) in the source distribution. * Disable all the options that allow user control over MANIFEST generation, including pruning, defaults, changing the filenames, etc. * Use the MANIFEST data (along with revision control info) not only for producing source distributions, but also to determine what files should be considered "package data", if the user passes an 'include_package_data=True' keyword to setup(). The net result would be a single source for what constitutes "the distribution contents", in the sense of files that are not directly part of the distutils build process. For files that are built automatically in some way but should be included in source distributions or as package data, you would still have to put them in MANIFEST.in. But anything that was under CVS or Subversion would be handled automatically, and you wouldn't have to duplicate data between MANIFEST.in and setup(package_data={...}). I'm also thinking that most of the MANIFEST logic could and should move to the Distribution class, since the data will be used by multiple commands. Thus, the sdist command could just ask the Distribution for the MANIFEST and get it, as would the commands that copy package data files to the build directory. I suspect the most controversial parts of this idea are: * Disabling all user control of MANIFEST * Forcibly including MANIFEST and MANIFEST.in in source distributions * Making MANIFEST be always platform-independent When Googling the issues around MANIFEST, I noticed that the idea of having MANIFEST or MANIFEST.in included automatically has been repeatedly shot down here over the years. However, if I followed the logic put forth on those occasions, I would never have implemented revision control support in the first place, so I guess if I'm in for a penny, I might as well be in for a pound, as they say. I couldn't find any argument one way or the other about the manifest-generation options, nor any reasons why MANIFEST needs to remain platform-specific, so I presume the options are just YAGNI and the format was just an implementation accident. Likewise, as far as I can tell there is no reason for *not* regenerating a MANIFEST whenever you need one, so the current behavior of only building one when MANIFEST.in changes or you use --force-manifest, seems like a premature optimization. Or maybe it wasn't an excessive optimization when the distutils were created, but it's not as if it's going to save you much time compared to the actual archive building process today. I'm thinking that basically --force-manifest would become a no-op in setuptools, in the sense that you won't be able to *stop* the MANIFEST from being built every single time. --manifest-only would still be possible. --manifest and --template would have to be rejected, however, because the standard name is needed for MANIFEST to be re-read when you build stuff from the produced sdist. --no-defaults would be ignored, except for a warning. If you don't want the defaults, you can always start your MANIFEST.in with an exclude pattern to exclude absolutely everything already included. There shouldn't be two ways to do the same thing, especially not one that you can use on the command line to mess things up in a non-repeatable fashion! Likewise --no-prune, because that's a similar recipe for disaster. A lot of these ideas are potential backward compatibility problems, so we'll have to see how they play out in setuptools before considering them for addition to the distutils. My guess, however, is that most prolific Python developers want to spend their time writing code, not writing and debugging MANIFEST.in files, and that fact has been responsible for a lot of setuptools uptake so far. I've been seeing a lot of projects that use setuptools for no apparent reason other than it makes writing the setup script a little easier, due to find_packages(), package_data, and the lack of need for a MANIFEST when source control is involved. These are qualities I'd like to extend further, even at the cost of some flexibility. Heck, most of the distutils' flaws lie in their extreme versatility. You can tell each individual command that it's using different build or distribution directories, for example, and in the process completely foul up your builds. What's more, every distutils tutorial may well end up giving people different instructions as to the "best" way to lay out a project directory. If there's ever a "distutils 2", it needs to become dictator-ware and tell you exactly what the One Obvious Way is. If everything *had* to be a particular way, then changing how the distutils work would actually be possible, whereas now, it's bloody hard to even figure out which of the nine billion ways to do it are actually in use. Okay, off the soapbox now. :) Does anybody see any issues with this that I'm missing, with respect to using the MANIFEST/FileList machinery to control sdist and package data, or my implementation plans for doing so? Thanks.

6 12

Re: [Distutils] [TurboGears] hardcoded paths in tg-admin.exe and easy_install.exe?
by Ian Bicking Nov. 16, 2005

Nov. 16, 2005

rtilley(a)vt.edu wrote: > Perhaps this has been mentioned already. So, forgive me if it has. > > I install Python into 'C:/Program Files/Python24' as I do not like to > clutter up the C:/ portion of my file system. After installing > turbogears and trying to begin the tutorial, I get the below error when > I do this 'tg-admin quickstart': > > Program: can't open file 'Files\Python24\python.exe' : [Error 2] No > such file or directory > > The same thing happens when I try to run easy_install.exe > > The .py versions of tg-admin and easy_install work fine... it's just > the .exe versions that crash. I have named this the 'hard-coded python > path bug'. > > Is this a know issue that will be fixed? > > Also, the instructions about adding 'C:\Python24' and > 'C:\Python24\Scripts' to the PATH variable should be adjusted to > wherever the user has actually installed Python ;) Sounds like a bug in .exe generator that it doesn't like spaces. -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org

2 2

subway setup.py failure
by Robin Becker Nov. 16, 2005

Nov. 16, 2005

I'm getting this strangness from the subway setup script. Installed C:\Python\Lib\site-packages\FormEncode-0.3-py2.4.egg Processing dependencies for FormEncode>=0.2.2 Searching for CherryPy>=2.1.0 Reading http://www.python.org/pypi/CherryPy/ Reading http://www.cherrypy.org Reading http://sourceforge.net/project/showfiles.php?group_id=56099 Best match: CherryPy 2.1.0-rc2 Downloading http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?downl… error: Unexpected HTML page found at http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0-rc2.tar.gz?downl… It seems that easy_install isn't picking up from SF right now and in addition it hasn't noticed that the latest version is now 2.1.0. I tried a manual installation of the cherrypy as follows C:\Tmp\subway_svn\trunk>\python\Scripts\easy_install "CherryPy==2.1.0" Searching for CherryPy==2.1.0 Reading http://www.python.org/pypi/CherryPy/ Reading http://www.cherrypy.org Reading http://sourceforge.net/project/showfiles.php?group_id=56099 Best match: CherryPy 2.1.0 Downloading http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0.win32.exe?downlo… error: Unexpected HTML page found at http://prdownloads.sourceforge.net/cherrypy/CherryPy-2.1.0.win32.exe?downlo… an it seems I have to try an exact download path to get things right eg C:\Tmp\subway_svn\trunk>\python\Scripts\easy_install http://heanet.dl.sourceforge.net/sourceforge/cherrypy/CherryPy-2.1. 0.tar.gz Downloading http://heanet.dl.sourceforge.net/sourceforge/cherrypy/CherryPy-2.1.0.tar.gz Processing CherryPy-2.1.0.tar.gz Running CherryPy-2.1.0\setup.py -q bdist_egg --dist-dir c:\temp\easy_install-uu1h8w\CherryPy-2.1.0\egg-dist-tmp-jkracy zip_safe flag not set; analyzing archive contents... cherrypy._cphttptools: module references __file__ cherrypy.lib.autoreload: module references __file__ cherrypy.lib.covercp: module references __file__ cherrypy.lib.cptools: module references __file__ cherrypy.lib.profiler: module references __file__ cherrypy.test.test: module references __file__ cherrypy.test.test_core: module references __file__ cherrypy.test.test_static_filter: module references __file__ cherrypy.tutorial.tut09_files: module references __file__ cherrypy.tutorial.tut10_http_errors: module references __file__ Adding CherryPy 2.1.0 to easy-install.pth file Installed C:\Python\Lib\site-packages\CherryPy-2.1.0-py2.4.egg Processing dependencies for CherryPy==2.1.0 -- Robin Becker

4 4

EntryPoint.require() when dist=None
by Ashley Walsh Nov. 16, 2005

Nov. 16, 2005

EntryPoint.require() fails when extras==() and dist==None because it still attempt to call dist.requires(). Should the method be changed to somthting like:: def require(self, env=None, installer=None): if self.extras and not self.dist: raise UnknownExtra("Can't require() without a distribution", self) if self.dist: map(working_set.add, working_set.resolve(self.dist.requires (self.extras),env,installer)) Cheers, Ashley

2 1