From pje at telecommunity.com Wed Aug 1 00:05:49 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 31 Jul 2007 18:05:49 -0400 Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> References: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> Message-ID: <20070731220328.38C443A40A7@sparrow.telecommunity.com> At 05:16 PM 7/31/2007 -0400, Stanley A. Klein wrote: >Robert Kern wrote: >Stanley A. Klein wrote: > > >> I tried to do something to fix the numpy distutils bdist_rpm.py (by > >> trying to follow what was done in install.py) but it didn't work and I > >> got an error message I didn't understand. > > >I'd like to help, but if you don't copy the exact error message here, I >can't. > >OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- >obviously not well -- to follow what was done in >numpy/distutils/commands/install.py: Please see my earlier message about "setup.py install --root --record" and try that -- it will give us the information to know whether there's even a problem with numpy's bdist_rpm, or if it's in numpy's install_lib or build_ext commands. There's no point in changing the commands around before that. From jim at zope.com Wed Aug 1 21:37:53 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 1 Aug 2007 15:37:53 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: <20070731162913.1125B3A40F7@sparrow.telecommunity.com> References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> <20070726194836.89D0B3A403D@sparrow.telecommunity.com> <46AEDB43.6020104@weitershausen.de> <20070731162913.1125B3A40F7@sparrow.telecommunity.com> Message-ID: On Jul 31, 2007, at 12:31 PM, Phillip J. Eby wrote: > At 08:48 AM 7/31/2007 +0200, Philipp von Weitershausen wrote: >> But isn't .egg-info completely recreatable through setup.py? > > Not if you're putting other data there, no. Chandler, for example, > has an i18n framework that stores message catalogs and other > localized resources there. Um. Why? That isn't egg meta data is it? Couldn't that just go in good old package data? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Wed Aug 1 21:52:46 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Aug 2007 15:52:46 -0400 Subject: [Distutils] Best practices on distributing documentation and examples with eggs? In-Reply-To: References: <46A854FA.3080906@enthought.com> <20070726155222.18CB43A40D7@sparrow.telecommunity.com> <660761AD-A57D-4D81-A0F5-E328043D8B34@python.org> <20070726194836.89D0B3A403D@sparrow.telecommunity.com> <46AEDB43.6020104@weitershausen.de> <20070731162913.1125B3A40F7@sparrow.telecommunity.com> Message-ID: <20070801195121.A723F3A40A4@sparrow.telecommunity.com> At 03:37 PM 8/1/2007 -0400, Jim Fulton wrote: >On Jul 31, 2007, at 12:31 PM, Phillip J. Eby wrote: > >>At 08:48 AM 7/31/2007 +0200, Philipp von Weitershausen wrote: >>>But isn't .egg-info completely recreatable through setup.py? >> >>Not if you're putting other data there, no. Chandler, for example, >>has an i18n framework that stores message catalogs and other >>localized resources there. > >Um. Why? > >That isn't egg meta data is it? Egg metadata is data that belongs to an egg as a whole, rather than to any specific package within the egg. > Couldn't that just go in good old package data? Not if the egg doesn't contain any packages, and it's possible to ship a set of localizations as an egg, without it containing any actual code. From sklein at cpcug.org Wed Aug 1 23:30:01 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 1 Aug 2007 17:30:01 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <20070731220328.38C443A40A7@sparrow.telecommunity.com> References: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> <20070731220328.38C443A40A7@sparrow.telecommunity.com> Message-ID: <4658.207.188.248.157.1186003801.squirrel@www.cpcug.org> On Tue, 2007-07-31 at 18:05 -0400, Phillip J. Eby wrote: > At 05:16 PM 7/31/2007 -0400, Stanley A. Klein wrote: > >Robert Kern wrote: > >Stanley A. Klein wrote: > > > > >> I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I > > >> got an error message I didn't understand. > > > > >I'd like to help, but if you don't copy the exact error message here, I > >can't. > > > >OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in > >numpy/distutils/commands/install.py: > > Please see my earlier message about "setup.py install --root > --record" and try that -- it will give us the information to know whether there's even a problem with numpy's bdist_rpm, or if it's in numpy's install_lib or build_ext commands. There's no point in changing the commands around before that. I ran it and think I have figured out what is happening. I first ran the build, using "python setup.py build" rather than the "env CFLAGS="$RPM_OPT_FLAGS" python setup.py build" that appears in the rpm spec file built by bdist_rpm. I don't know what difference that makes. The build looked OK. I then created a buildroot directory and ran "python setup.py install --root=build/buildroot --record=INSTALLED_FILES". The install looked OK as well. It included the .so files and also built the .pyo and .pyc files for the .py files that are used during kiva execution and resulted from the [install] optimize=1 in setup.cfg. I saved the buildroot directory and INSTALLED_FILES. I then cleared the build directory and ran "python setup.py bdist_rpm". It gave me the unpackaged files error as before. I looked at exactly what files were unpackaged. They turned out to include two kinds of files: (1) some .pyc and .pyo files for .py files in example and test directories, and (2) a pair of .pyc and .pyo files for the __init__.py at the top level of the enthought directory that contains the statement for proper processing of namespace packages. I'm sure these .pyc and .pyo files are coming from brp-python-bytecompile that Fedora rpmbuild runs without requiring __init__.py files in the directories it processes. The workaround for the example and test directories is to remove them from the package. Under the Unix File Hierarchy Standard, they don't belong in /usr/lib/python/site-packages but in /usr/share/packagename+ or /usr/share/doc/packagename+ (where the "+" indicates adding version information). I had to do this with another enthought component, and for that I put them in a directory intended for building a package to install them in the right location per the FHS. I don't know what to do about the .pyc and .pyo files for the __init__.py. It looks like what might be happening is that the rpm spec file is being built by a combination of the distutils and numpy distutils functions (which differ from setuptools in the scripts they put in the spec file, including the env statement build difference shown above and omitting " --single-version-externally-managed" from the install options). Then it looks like the build might be executed by the numpy distutils but the install is being done by setuptools. The install (both what I executed alone and when done by bdist_rpm) includes the egg-info directory and the nspkg.pth file. BTW, differencing the bdist_rpm INSTALLED_FILES and my earlier test INSTALLED_FILES showed the only difference being the naming of the files in the egg-info and nspkg.pth files. I think this clarifies what is happening, although I'm not sure how to fix the issue with the __init__.py file. Many thanks for insisting that I run the test you suggested. I hope this helps make the issues clear. Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070801/efc2ed5b/attachment.htm From sklein at cpcug.org Thu Aug 2 18:11:06 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 2 Aug 2007 12:11:06 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <20070731220328.38C443A40A7@sparrow.telecommunity.com> References: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> <20070731220328.38C443A40A7@sparrow.telecommunity.com> Message-ID: <5087.207.188.248.157.1186071066.squirrel@www.cpcug.org> Robert Kern wrote: >Stanley A. Klein wrote: >> Robert Kern wrote: >> Stanley A. Klein wrote: >> >>>> I tried to do something to fix the numpy distutils bdist_rpm.py (by >>>> trying to follow what was done in install.py) but it didn't work and I >>>> got an error message I didn't understand. >> >>> I'd like to help, but if you don't copy the exact error message here, I >> can't. >> >> OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- >> obviously not well -- to follow what was done in >> numpy/distutils/commands/install.py: >> >> import os >> import sys >> >> if 'setuptools' in sys.modules: >> import setuptools.command.bdist_rpm as old_bdist_rpm >> class bdist_rpm(old_bdist_rpm): >> pass >> >> else: >> from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm > >Note the difference between these two lines. You imported a module, not the >class inside it. Try this instead: > >if 'setuptools' in sys.modules: > from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm >else: > from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm > Thanks. I tried it and it ran. I'm still having problems getting things to work, but as you can see from my previous email I've closed in on the issues, that now appear to involve the packaging of documentation, tests, and examples, and the handling of the namespace package __init__.py. Right now, I think my problems involve svn, relating to my moving the tests and examples to a different directory but somehow (probably because setuptools looks at svn information) having files I moved out of the enthought.kiva directory still in the build and creating problems, or files not in the current version expected for the build and found missing. I didn't follow proper svn procedure in moving files out, and I think that is coming back to bite me, although what I did seemed to work with another component. My best bet at this point is to re-checkout the enthought.branches directory and do it right from an svn viewpoint. Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070802/2fb1f9f1/attachment.html From sklein at cpcug.org Thu Aug 2 20:06:36 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 2 Aug 2007 14:06:36 -0400 (EDT) Subject: [Distutils] Best practices on distributing documentation and examples with eggs? Message-ID: <42278.207.188.248.157.1186077996.squirrel@www.cpcug.org> On Thu, July 26, 2007 9:50 pm, Phillip J. Eby wrote: >At 02:13 PM 7/26/2007 -0500, Dave Peterson wrote: >>In further discussions at Enthought, we're now thinking that creating a 'docs' dir as a top-level under python itself makes sense, and then the installation of eggs could copy docs and/or examples there in a manner similar to how it handles scripts into the platforms scripts dir. It would make sense that they should be put in dirs named with a format like - so that the user could differentiate the docs for each project they installed. If this was done, it seems like it wouldn't be too hard for Stan, and other rpm builders, to either create symbolic links or copy these to /usr/share/doc in order to maintain compatibility with the LSB. >This isn't a bad idea, although I think that there are some hurdles that need to be worked out. My general thought is that, were >easy_install to support documentation, it would need to take an >option to determine the documentation base directory, and have a --no-docs install option. >One issue is that under most RPM systems, package names are more like 'python-foobar-1.2', and 'foobar' may not even be the exact distutils name of the package. So, installing the docs to the correct location in the general case may be more complex. Note, however, that >bdist_rpm offers a --doc-files option, which of course can be >configured via setup.cfg. I have no idea how it works, though, so don't ask me. :) The practice of changing the name of the whatever package to python-whatever may be distro dependent. Also (thankfully), python-whatever is installed in site-packages/whatever and not in site-packages/python-whatever. It was not always that way. Back at something like RedHat 7, they did change the names and the installation locations. If you had another program that imported the package, you sometimes had to go in and change the code to match their changed name. I can understand the reasons of the distros and repositories for doing this (keeping all the python or other language packages together in the list) but that does not make it less annoying. In looking to see if a package is in the repository, you have to look in various places beyond what the original developer/maintainer of the package calls it. Also, I'm not sure of the split between /usr/share/whatever and usr/share/doc/whatever for things like docs, tests, examples, and other miscellaneous files. It tends to be at the whim of the packager. In terms of what bdist_rpm does, it seems to be in the preparation of the rpm spec file. The spec file contains scripts (some of which point to macros in /usr/lib/rpm/macros on a Fedora system) and bdist_rpm seems to build the scripts and execute them using rpm-build. The build and install scripts do "python setup.py build" or "python setup.py install" with whatever options are specified. I think the actual location of the files may be governed by the %files section of the spec, but I will have to spend some time looking at other spec files to be sure that is where it happens. Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070802/c75904a6/attachment.html From python at venix.com Sat Aug 4 02:29:49 2007 From: python at venix.com (Lloyd Kvam) Date: Fri, 03 Aug 2007 20:29:49 -0400 Subject: [Distutils] Windows web based installation of multiple packages Message-ID: <1186187389.4695.21.camel@localhost.localdomain> (I normally work from Linux, but have a test WinXP computer and my old WinNT4 with Visual Studio 6 should that turn out to be useful.) It is quite easy to use the bdist_wininst command to create a Windows installer executable. A browser click on a link to the .EXE file allows for software installation, but easy_install is not run and the packages listed in setup.install_requires are not installed. Is there a way to force the windows installer to run easy_install? If I were to create an egg, is there a content-type heading that would cause easy_install to process the egg? (much like application/vnd.ms-excel) I could not find an easy_install entry in the registry in my test WinXP computer after installing setuptools. I thought one of these approaches would be fairly easy, but googling and experimenting have gotten me nowhere. -- Lloyd Kvam Venix Corp From keith at dartworks.biz Sun Aug 5 09:52:49 2007 From: keith at dartworks.biz (Keith Dart) Date: Sun, 5 Aug 2007 00:52:49 -0700 Subject: [Distutils] bug in svn detection Message-ID: <20070805005249.2bba2430@tinker.homenetwork> Greetings. I would like to report a bug in setuptools here, since I can't find a bug tracking system for it anywhere. 598 $ sudo python setup.py install running install running bdist_egg exceptions.ValueError : max() arg is an empty sequence > /usr/lib/python2.4/site-packages/setuptools/command/egg_info.py(224) in get_svn_revision() localrev = max([int(d[9]) for d in data if len(d)>9 and d[9]]) Debug:15/15> l 214 continue # no sense walking uncontrolled subdirs 215 dirs.remove('.svn') 216 f = open(os.path.join(base,'.svn','entries')) 217 data = f.read() 218 f.close() 219 220 if data.startswith('8'): 221 data = map(str.splitlines,data.split('\n\x0c\n')) 222 del data[0][0] # get rid of the '8' 223 dirurl = data[0][3] 224 -> localrev = max([int(d[9]) for d in data if len(d)>9 and d[9]]) 225 elif data.startswith(' show get_svn_revision ( self = , ) Compiled locals: files = ['mkpydocindex'] revre = <_sre.SRE_Pattern object at 0xb7cf4458> base_url = 'https://pycopia.googlecode.com/svn/trunk/vim/' localrev = 11 revision = 53 dirs = [] base = './bin' data = [['', 'dir', '0', 'https://pycopia.googlecode.com/svn/trunk/vim/bin', 'https://pycopia.googlecode.com/svn', 'add', ...], ['mkpydocindex', 'file', '', '', '', 'add', ...], []] d = [] f = m = '*** undefined ***' _[1] = '*** undefined ***' dirurl = 'https://pycopia.googlecode.com/svn/trunk/vim/bin' urlre = <_sre.SRE_Pattern object at 0xb7ba49a0> 601 !$ svn --version svn, version 1.4.4 (r25188) compiled Jun 29 2007, 12:11:09 (However, the repository may have been created with an older version of subversion). I guess Phillip would be most interested in this: 51900 phillip.eby 51900 phillip.eby if data.startswith('8'): 51900 phillip.eby data = map(str.splitlines,data.split('\n\x0c\n')) 51900 phillip.eby del data[0][0] # get rid of the '8' 51900 phillip.eby dirurl = data[0][3] 56277 phillip.eby localrev = max([int(d[9]) for d in data if len(d)>9 and d[9]]+[0]) 51900 phillip.eby elif data.startswith(' public key: ID: 19017044 ===================================================================== From pje at telecommunity.com Mon Aug 6 00:31:32 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Aug 2007 18:31:32 -0400 Subject: [Distutils] bug in svn detection In-Reply-To: <20070805005249.2bba2430@tinker.homenetwork> References: <20070805005249.2bba2430@tinker.homenetwork> Message-ID: <20070805223407.21A843A40A8@sparrow.telecommunity.com> At 12:52 AM 8/5/2007 -0700, Keith Dart wrote: >I guess Phillip would be most interested in this: > > 51900 phillip.eby > 51900 phillip.eby if data.startswith('8'): > 51900 phillip.eby data = > map(str.splitlines,data.split('\n\x0c\n')) > 51900 phillip.eby del data[0][0] # get rid of the '8' > 51900 phillip.eby dirurl = data[0][3] > 56277 phillip.eby localrev = max([int(d[9]) for d > in data if len(d)>9 and d[9]]+[0]) > 51900 phillip.eby elif data.startswith(' 51900 phillip.eby dirurl = > urlre.search(data).group(1) # get repository URL > 56277 phillip.eby localrev = max([int(m.group(1)) > for m in revre.finditer(data)]+[0]) I don't see why; the fixes are already in setuptools SVN repository, as you can see here in r56277. AFAIK, this code is in setuptools 0.6c6, i.e., the current release. From eucci.group at gmail.com Mon Aug 6 15:57:50 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Mon, 6 Aug 2007 07:57:50 -0600 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic Message-ID: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> Hi. I'm trying to update our internal code to use distutils/setuptools at long last. We work heavily in Zope 3, so our packages contain many "data" files (ZCML, Page Templates, Javascript, CSS, etc). All of our internal code is maintained in CVS. I noticed that using ``include_package_data = True`` was having little or no effect: at most, only data files in the root directory might be included. This seems to be related to our directories being listed in `CVS/Entries.Log` instead of `CVS/Entries`, and setuptools doesn't read this file. I don't know why our directories are in this file: these are fresh checkouts (retrieved over the network using :ext: and SSH) with no local changes having yet been applied. Apparently programs that are reading 'Entries' should also read 'Entries.Log', which is slightly different from Entries. I have more details below. Is there any particular reason why this file isn't being read? I'd be surprised if I'm the only one encountering this issue. A basic fix appears to be easy: add another `re_finder` for `CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. I'm sure I could add something like this with an entry-point and our own egg, but it seems ridiculously small for us to maintain internally. Details ======= I traced this issue down to setuptools/command/sdist.py and the default set of finders:: finders = [ (convert_path('CVS/Entries'), re_finder(re.compile(r"^\w?/([^/]+)/", re.M))), (convert_path('.svn/entries'), entries_finder), (convert_path('.svn/dir-props'), externals_finder), (convert_path('.svn/dir-prop-base'), externals_finder), # svn 1.4 ] That looked right to me. Then I looked in a CVS directory in a couple of different packages and looked at Entries:: CVS> cat Entries /__init__.py/1.1/Thu Aug 18 23:29:21 2005//Tfixup-branch /addhelp.py/1.1.2.2/Wed Dec 20 22:55:09 2006//Tfixup-branch /configure.zcml/1.5.2.10/Wed Jul 18 19:32:51 2007//Tfixup-branch # .... (more lines - no directories) /tests.py/1.1.2.2/Thu Jan 18 22:10:50 2007//Tfixup-branch /traversing.py/1.1.2.2/Tue Mar 20 23:29:36 2007//Tfixup-branch /valuelib.py/1.1.2.3/Fri Jul 27 20:26:38 2007//Tfixup-branch D Hmmm, no directories... Then I noticed an "Entries.Log" file. I looked at that:: CVS> cat Entries.Log A D/authentication//// A D/batching//// # ... (more lines, all directories, all lines beginning with ``A D``) A D/undo//// Whoa! Here are all of the directories! I'm not sure why these are in there - this is showing up in a fresh code checkout, and it's showing up regardless of whether the checked out code is a branch or not. This is showing up on Mac OS X with the CVS that comes with the development tools package. CVS> cvs --version Concurrent Versions System (CVS) 1.11.18 (client/server) Our server has ``1.11.1p1``. I don't know if these version numbers mean anything in relation to this issue. Does anybody else? Anyways, the CVS manual says this about Entries.Log:: > Programs which are reading the 'Entries' file should also check > for 'Entries.Log'...The format of a line in 'Entries.Log' is > a single character command followed by a space followed by a line in > the format specified for a line in 'Entries'. The single character > command is 'A' to indicate that the entry is being added, 'R' > to indicate that the entry is being removed, or any other character to > indicate that the entire line in 'Entries.Log' should be silently > ignored (for future expansion). http://www.network-theory.co.uk/docs/cvsmanual/Workingdirectorystorage.html So, it looks like a decent solution would be to add another `re_finder` in setuptools' `sdist.py` module as mentioned above that took the Entries.Log file and format into account.... Yes? Thanks, Jeff Shell From mhammond at skippinet.com.au Mon Aug 6 16:08:09 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Mon, 6 Aug 2007 09:08:09 -0500 Subject: [Distutils] Windows web based installation of multiple packages In-Reply-To: <1186187389.4695.21.camel@localhost.localdomain> References: <1186187389.4695.21.camel@localhost.localdomain> Message-ID: <01fb01c7d833$38045c10$6c01a8c0@bobcat> > It is quite easy to use the bdist_wininst command to create a > Windows installer executable. A browser click on a link to > the .EXE file allows for software installation, but > easy_install is not run and the packages listed in > setup.install_requires are not installed. Is there a way to > force the windows installer to run easy_install? > > If I were to create an egg, is there a content-type heading > that would cause easy_install to process the egg? > (much like application/vnd.ms-excel) I could not find > an easy_install entry in the registry in my test WinXP > computer after installing setuptools. > > I thought one of these approaches would be fairly easy, but > googling and experimenting have gotten me nowhere. bdist_wininst is designed to install a python package, not to install a python "application" which has external dependencies. You may like to consider py2exe, which is designed to create a stand-alone executable from your python app - but even then, eggs etc are not processed - you are expected to have all relevant packages installed on your dev box, and they are packaged inside the app. Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2002 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070806/0e8e3f21/attachment.bin From pje at telecommunity.com Mon Aug 6 17:43:18 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Aug 2007 11:43:18 -0400 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.co m> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> Message-ID: <20070806154912.C3DC33A403D@sparrow.telecommunity.com> At 07:57 AM 8/6/2007 -0600, Jeff Shell wrote: >A basic fix appears to be easy: add another `re_finder` for >`CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. I'd add this, but since I don't have data to test it on and you apparently do, please send a tested patch and I'll add it to the next version. >I'm sure I >could add something like this with an entry-point and our own egg, but >it seems ridiculously small for us to maintain internally. Of course; it should be in setuptools. It's just been years since I've used CVS, and I've never encountered this issue. From pje at telecommunity.com Mon Aug 6 17:55:00 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Aug 2007 11:55:00 -0400 Subject: [Distutils] bug in svn detection In-Reply-To: <20070805184614.6c01ff58@psyche.corp.google.com> References: <20070805005249.2bba2430@tinker.homenetwork> <20070805223407.21A843A40A8@sparrow.telecommunity.com> <20070805184614.6c01ff58@psyche.corp.google.com> Message-ID: <20070806155234.794A33A403D@sparrow.telecommunity.com> At 06:46 PM 8/5/2007 -0700, Keith Dart ??? wrote: >Phillip J. Eby wrote the following on 2007-08-05 at 18:31 PDT: >=== > > I don't see why; the fixes are already in setuptools SVN repository, > > as you can see here in r56277. AFAIK, this code is in setuptools > > 0.6c6, i.e., the current release. > >=== > >Oh, ok. It's fixed already. I didn't know you knew about it. Yes, I am >using 0.6c6, since that is the recommended release (and what my Linux >distribution packages). So, the only fix then is to install setuptools >from SVN HEAD? It would appear so; I guess I'm wrong about this being in 0.6c6; I just looked at my checkout, which is actually an 0.6c7dev version. Use "easy_install setuptools==dev06" to get the same version as I'm using. From eucci.group at gmail.com Mon Aug 6 19:31:00 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Mon, 6 Aug 2007 11:31:00 -0600 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <20070806154912.C3DC33A403D@sparrow.telecommunity.com> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> <20070806154912.C3DC33A403D@sparrow.telecommunity.com> Message-ID: <88d0d31b0708061031g39213676wf26fea0f2904aa19@mail.gmail.com> On 8/6/07, Phillip J. Eby wrote: > At 07:57 AM 8/6/2007 -0600, Jeff Shell wrote: > >A basic fix appears to be easy: add another `re_finder` for > >`CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. > > I'd add this, but since I don't have data to test it on and you > apparently do, please send a tested patch and I'll add it to the next version. By "tested patch", do you want a unit test to accompany the patch? Or do you just want it to have been tested by me and compared against the current release? I ask because there doesn't seem to be any test coverage of the "walk_revcontrol()" functionality, and writing such tests that depend on directory structures and files and bla bla bla is quite a pain as I'm sure you know. (Especially testing for revision control based files that are meant to be example files that also might be under, um, revision control). In any case, I think I have such a patch. Where to send? > >I'm sure I > >could add something like this with an entry-point and our own egg, but > >it seems ridiculously small for us to maintain internally. > > Of course; it should be in setuptools. It's just been years since > I've used CVS, and I've never encountered this issue. I'm still not sure why I'm seeing it. I have verified that it only seems to happen on a fresh checkout. If I run ``cvs up -dP`` (and a regular ``cvs up`` would probably do fine) right after checkout, almost all of those Entries.Log files go away. The ones that remain specifically target emptied/removed directories and have matching lines like:: A D/MochiKit//// R D/MochiKit//// For "add mochikit; remove mochikit". So the patch might not really be needed..? If I change our release strategy to the following, then I get all of my package data just fine in the 0.6c6 release. > cvs checkout -d ourlib Frameworks/ourlib > cd ourlib > cvs update -dP > python setup.py bdist_egg With my patched `sdist.py`, however, I can skip the `update` line and get the same desired outcome. -- Jeff Shell From pje at telecommunity.com Mon Aug 6 19:51:21 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Aug 2007 13:51:21 -0400 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <88d0d31b0708061031g39213676wf26fea0f2904aa19@mail.gmail.co m> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> <20070806154912.C3DC33A403D@sparrow.telecommunity.com> <88d0d31b0708061031g39213676wf26fea0f2904aa19@mail.gmail.com> Message-ID: <20070806174857.858DD3A40AA@sparrow.telecommunity.com> At 11:31 AM 8/6/2007 -0600, Jeff Shell wrote: >On 8/6/07, Phillip J. Eby wrote: > > At 07:57 AM 8/6/2007 -0600, Jeff Shell wrote: > > >A basic fix appears to be easy: add another `re_finder` for > > >`CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. > > > > I'd add this, but since I don't have data to test it on and you > > apparently do, please send a tested patch and I'll add it to the > next version. > >By "tested patch", do you want a unit test to accompany the patch? Or >do you just want it to have been tested by me and compared against the >current release? The latter. >In any case, I think I have such a patch. Where to send? Me, the list, or both. :) From jim at zope.com Mon Aug 6 19:56:36 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 6 Aug 2007 13:56:36 -0400 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <20070806154912.C3DC33A403D@sparrow.telecommunity.com> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> <20070806154912.C3DC33A403D@sparrow.telecommunity.com> Message-ID: <4F4F4B08-C59B-43F6-B666-0942878E2C3D@zope.com> On Aug 6, 2007, at 11:43 AM, Phillip J. Eby wrote: > At 07:57 AM 8/6/2007 -0600, Jeff Shell wrote: >> A basic fix appears to be easy: add another `re_finder` for >> `CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. > > I'd add this, but since I don't have data to test it on and you > apparently do, please send a tested patch and I'll add it to the > next version. The patch should have an accompanying test to prevent a regression. It should be possible to provide the necessary file structure for a test. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Aug 6 19:57:26 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 6 Aug 2007 13:57:26 -0400 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <20070806174857.858DD3A40AA@sparrow.telecommunity.com> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> <20070806154912.C3DC33A403D@sparrow.telecommunity.com> <88d0d31b0708061031g39213676wf26fea0f2904aa19@mail.gmail.com> <20070806174857.858DD3A40AA@sparrow.telecommunity.com> Message-ID: <6ED393FE-E1AB-47E0-80AA-609200ACEE80@zope.com> On Aug 6, 2007, at 1:51 PM, Phillip J. Eby wrote: > At 11:31 AM 8/6/2007 -0600, Jeff Shell wrote: >> On 8/6/07, Phillip J. Eby wrote: >>> At 07:57 AM 8/6/2007 -0600, Jeff Shell wrote: >>>> A basic fix appears to be easy: add another `re_finder` for >>>> `CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. >>> >>> I'd add this, but since I don't have data to test it on and you >>> apparently do, please send a tested patch and I'll add it to the >> next version. >> >> By "tested patch", do you want a unit test to accompany the patch? Or >> do you just want it to have been tested by me and compared against >> the >> current release? > > The latter. This would be a mistake. The former. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From eucci.group at gmail.com Mon Aug 6 19:58:45 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Mon, 6 Aug 2007 11:58:45 -0600 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <20070806174857.858DD3A40AA@sparrow.telecommunity.com> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> <20070806154912.C3DC33A403D@sparrow.telecommunity.com> <88d0d31b0708061031g39213676wf26fea0f2904aa19@mail.gmail.com> <20070806174857.858DD3A40AA@sparrow.telecommunity.com> Message-ID: <88d0d31b0708061058y3af54c4xc9b252e38198cad6@mail.gmail.com> On 8/6/07, Phillip J. Eby wrote: > At 11:31 AM 8/6/2007 -0600, Jeff Shell wrote: > >In any case, I think I have such a patch. Where to send? > > Me, the list, or both. :) Excellent. Diff's attached. It might make the old `re_finder` function obsolete. -- Jeff Shell -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools06_cvs_entrylog_fix.diff Type: application/octet-stream Size: 2832 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070806/390c13ff/attachment.obj From eucci.group at gmail.com Mon Aug 6 20:20:03 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Mon, 6 Aug 2007 12:20:03 -0600 Subject: [Distutils] CVS/Entries.Log ignored by setuptools default CVS file-finder logic In-Reply-To: <6ED393FE-E1AB-47E0-80AA-609200ACEE80@zope.com> References: <88d0d31b0708060657s2bcbd2c0i8f79f0563492df12@mail.gmail.com> <20070806154912.C3DC33A403D@sparrow.telecommunity.com> <88d0d31b0708061031g39213676wf26fea0f2904aa19@mail.gmail.com> <20070806174857.858DD3A40AA@sparrow.telecommunity.com> <6ED393FE-E1AB-47E0-80AA-609200ACEE80@zope.com> Message-ID: <88d0d31b0708061120g6697db06tbc78580ce39fd86c@mail.gmail.com> > On Aug 6, 2007, at 1:51 PM, Phillip J. Eby wrote: > > > At 11:31 AM 8/6/2007 -0600, Jeff Shell wrote: > >> On 8/6/07, Phillip J. Eby wrote: > >>> At 07:57 AM 8/6/2007 -0600, Jeff Shell wrote: > >>>> A basic fix appears to be easy: add another `re_finder` for > >>>> `CVS/Entries.Log` with a regex like ``^A\s\w?/([^/]+)/``. > >>> > >>> I'd add this, but since I don't have data to test it on and you > >>> apparently do, please send a tested patch and I'll add it to the > >> next version. > >> > >> By "tested patch", do you want a unit test to accompany the patch? Or > >> do you just want it to have been tested by me and compared against > >> the > >> current release? > > > > The latter. > > This would be a mistake. The former. :) I agree. However, there are no existing tests for this feature, so there's no place to plug in. It would require generating fake packages with fake CVS (and complementary tests should do subversion) files and directories and yada yada yada. I looked and looked for something I might use as a starting point. Since I know next to nothing about the overall structure of setuptools and distutils, I feel rather unqualified to write such a test completely from scratch in the time I have available. -- Jeff Shell From sklein at cpcug.org Mon Aug 6 21:47:38 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 6 Aug 2007 15:47:38 -0400 (EDT) Subject: [Distutils] How can I get setuptools to include files beyond what find_packages() finds? In-Reply-To: <5087.207.188.248.157.1186071066.squirrel@www.cpcug.org> References: <4235.207.188.248.157.1185916568.squirrel@www.cpcug.org> <20070731220328.38C443A40A7@sparrow.telecommunity.com> <5087.207.188.248.157.1186071066.squirrel@www.cpcug.org> Message-ID: <8676.207.188.248.157.1186429658.squirrel@www.cpcug.org> On Thu, 2007-08-02 at 12:11 -0400, Stanley A. Klein wrote: > Robert Kern wrote: > >Stanley A. Klein wrote: > >> Robert Kern wrote: > >> Stanley A. Klein wrote: > >> > >>>> I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I > >>>> got an error message I didn't understand. > >> > >>> I'd like to help, but if you don't copy the exact error message here, I > >> can't. > >> > >> OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- > >> obviously not well -- to follow what was done in > >> numpy/distutils/commands/install.py: > >> > >> import os > >> import sys > >> > >> if 'setuptools' in sys.modules: > >> import setuptools.command.bdist_rpm as old_bdist_rpm > >> class bdist_rpm(old_bdist_rpm): > >> pass > >> > >> else: > >> from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm > > > >Note the difference between these two lines. You imported a module, not the > >class inside it. Try this instead: > > > >if 'setuptools' in sys.modules: > > from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm > >else: > > from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm > > > > Thanks. > > I tried it and it ran. > > I'm still having problems getting things to work, but as you can see from > my previous email I've closed in on the issues, that now appear to involve > the packaging of documentation, tests, and examples, and the handling of the namespace package __init__.py. > > Right now, I think my problems involve svn, relating to my moving the tests and examples to a different directory but somehow (probably because > setuptools looks at svn information) having files I moved out of the enthought.kiva directory still in the build and creating problems, or files not in the current version expected for the build and found missing. > > I didn't follow proper svn procedure in moving files out, and I think that > is coming back to bite me, although what I did seemed to work with another > component. My best bet at this point is to re-checkout the > enthought.branches directory and do it right from an svn viewpoint. > I may have spoken too soon about it working. I'm now getting a completely incomprehensible error during the build step. If I independently run "python setup.py build" it works. If I run "python setup.py bdist_rpm" it creates an rpm spec file with substantially the same statement for the build and I get the traceback below. If I make the build statement exactly the same by specifying --no-rpm-opt-flags the behavior doesn't change. The traceback is (picking up slightly earlier in the output): ------------ adding 'enthought/kiva/agg/freetype2/src/sfnt/sfnt.c' to sources. adding 'enthought/kiva/agg/freetype2/src/smooth/smooth.c' to sources. adding 'enthought/kiva/agg/freetype2/src/truetype/truetype.c' to sources. adding 'enthought/kiva/agg/freetype2/src/type1/type1.c' to sources. adding 'enthought/kiva/agg/freetype2/src/type42/type42.c' to sources. adding 'enthought/kiva/agg/freetype2/src/winfonts/winfnt.c' to sources. adding 'enthought/kiva/agg/freetype2/src/gzip/ftgzip.c' to sources. building library "agg24_src" sources building library "kiva_src" sources building extension "enthought.kiva.agg._agg" sources Traceback (most recent call last): File "setup.py", line 117, in ? version = '2.0b1', File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 87, in run self.build_sources() File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 216, in build_extension_sources sources = self.swig_sources(sources, ext) File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 564, in swig_sources raise ValueError("%r missing" % (target_file,)) ValueError: 'agg_wrap.c' missing error: Bad exit status from /var/tmp/rpm-tmp.37529 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.37529 (%build) error: command 'rpmbuild' failed with exit status 1 ----------------------------- The problem is that when the build is done under the rpmbuild spec build script (which is the same as the command line used for an ordinary build), somehow it wants to look for a file agg_wrap.c. There is no file agg_wrap.c. There *is* a file agg_wrap.cpp, that it properly finds and uses when running "python setup.py build". I've looked at the numpy distutils code and can't imagine what would cause such a weird problem. Any suggestions? Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070806/9d00434d/attachment.htm From eucci.group at gmail.com Wed Aug 8 03:23:35 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Tue, 7 Aug 2007 19:23:35 -0600 Subject: [Distutils] setuptools, eggs, and setup.py in the root of a package - how? Message-ID: <88d0d31b0708071823x6b437d77kaaa5f3563a2cce8b@mail.gmail.com> So we've never done much with eggs or distutils up until now at my company. We've generally worked in Zope. We generally check out Python packages directly into a directory where Zope would automatically pick them up. So we've never gotten into any convention of having a layout that I commonly see these days: distroot - setup.py + src + ourlib - __init__.py - etc... Fearing that we'd have to dramatically change our source code around, and shake up existing checkouts, I've avoided trying to add any distutils/setuptools support to these packages. But I was told recently that I could have `setup.py` in the package root (`ourlib` in the example above). I've seen this work in a plain distutils system (Chris McDonough's BuildIt), but I've had a nightmare of a time getting it to work with setuptools and zc.buildout and on and on. Some code lies below that tries to explain the dilemma and shows my best guess efforts so far. Has anyone had any success with this layout? I was trying to use this as a 'develop' egg(?) using `zc.buildout`, and it doesn't work. In a generated 'virtual-python' kind of thing, it sets up this in sys.path (when specifying ``develop=src/ourlib`` in `buildout.cfg`): sys.path[0:0] = [ '/Users/jshell/Documents/Programming/eggify/src/fdlib', ... This seems like an uphill battle. Any tips? My fallback plan is to have a separate 'releases' area of our repository which houses setup.py and such for each of these packages along with a script to check out the main source before building an distribution or using as a develop-egg-thing. Is there anything, specific to this problem, that I can be doing better with this setup.py? from setuptools import setup, find_packages def list_packages(): packages = [ 'ourlib.%s' % subpackage for subpackage in find_packages() ] packages.insert(0, 'ourlib') return packages setup( name = 'ourlib', version = "1.0a1", packages = list_packages(), package_dir = {'ourlib': '.'}, include_package_data = True, install_requires=['setuptools'], extras_require = dict( test = ['zope.testing'], ), zip_safe = False, ) Without that 'list_packages', the results of `find_packages` would return 'toolkit' instead of 'ourlib.toolkit': the package head that would normally be there is off. Thanks, Jeff Shell From pje at telecommunity.com Wed Aug 8 03:51:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Aug 2007 21:51:08 -0400 Subject: [Distutils] setuptools, eggs, and setup.py in the root of a package - how? In-Reply-To: <88d0d31b0708071823x6b437d77kaaa5f3563a2cce8b@mail.gmail.co m> References: <88d0d31b0708071823x6b437d77kaaa5f3563a2cce8b@mail.gmail.com> Message-ID: <20070808014842.7EF903A40A4@sparrow.telecommunity.com> At 07:23 PM 8/7/2007 -0600, Jeff Shell wrote: >So we've never done much with eggs or distutils up until now at my >company. We've generally worked in Zope. We generally check out Python >packages directly into a directory where Zope would automatically pick >them up. So we've never gotten into any convention of having a layout >that I commonly see these days: > > distroot > - setup.py > + src > + ourlib > - __init__.py > - etc... > >Fearing that we'd have to dramatically change our source code around, >and shake up existing checkouts, I've avoided trying to add any >distutils/setuptools support to these packages. But I was told >recently that I could have `setup.py` in the package root (`ourlib` in >the example above). You can, but putting it one level higher (e.g. 'src') would be much better. Putting it inside the package means you can't get the "develop" or "test" commands to work correctly... as you mention below. > I've seen this work in a plain distutils system >(Chris McDonough's BuildIt), but I've had a nightmare of a time >getting it to work with setuptools and zc.buildout and on and on. > >Some code lies below that tries to explain the dilemma and shows my >best guess efforts so far. Has anyone had any success with this >layout? > >I was trying to use this as a 'develop' egg(?) using `zc.buildout`, >and it doesn't work. In a generated 'virtual-python' kind of thing, it >sets up this in sys.path (when specifying ``develop=src/ourlib`` in >`buildout.cfg`): > > sys.path[0:0] = [ > '/Users/jshell/Documents/Programming/eggify/src/fdlib', > ... > >This seems like an uphill battle. Any tips? "develop" won't work unless your project directory contains packages; i.e., setup.py has to be in "src" for this to work. That's because some directory needs to get put in sys.path, that has both the .egg-info/ dir and the actual packages as subdirectories. From sylvain at itaapy.com Wed Aug 8 13:32:44 2007 From: sylvain at itaapy.com (Sylvain Taverne) Date: Wed, 08 Aug 2007 13:32:44 +0200 Subject: [Distutils] One windows installer with PIL, docutils and itools for python 2.5 Message-ID: <46B9A9DC.6090106@itaapy.com> Hello, We need to make an executable installer for MS Windows, we guess using bdist_wininst. The installer should package in one single installation on python 2.5 PIL, docutils and itools. We should rather not compile PIL on windows (it's big) but use perhaps the binary distribution for windows: http://effbot.org/downloads/PIL-1.1.6.win32-py2.5.exe Thank's -- Sylvain From python at venix.com Wed Aug 8 16:32:54 2007 From: python at venix.com (Lloyd Kvam) Date: Wed, 08 Aug 2007 10:32:54 -0400 Subject: [Distutils] Windows web based installation of multiple packages In-Reply-To: <01fb01c7d833$38045c10$6c01a8c0@bobcat> References: <1186187389.4695.21.camel@localhost.localdomain> <01fb01c7d833$38045c10$6c01a8c0@bobcat> Message-ID: <1186583574.24047.96.camel@localhost.localdomain> On Mon, 2007-08-06 at 09:08 -0500, Mark Hammond wrote: > > It is quite easy to use the bdist_wininst command to create a > > Windows installer executable. A browser click on a link to > > the .EXE file allows for software installation, but > > easy_install is not run and the packages listed in > > setup.install_requires are not installed. Is there a way to > > force the windows installer to run easy_install? > > > > If I were to create an egg, is there a content-type heading > > that would cause easy_install to process the egg? > > (much like application/vnd.ms-excel) I could not find > > an easy_install entry in the registry in my test WinXP > > computer after installing setuptools. > > > > I thought one of these approaches would be fairly easy, but > > googling and experimenting have gotten me nowhere. > > bdist_wininst is designed to install a python package, not to install a > python "application" which has external dependencies. You may like to > consider py2exe, which is designed to create a stand-alone executable from > your python app - but even then, eggs etc are not processed - you are > expected to have all relevant packages installed on your dev box, and they > are packaged inside the app. Thanks for the reply. I have deadlines to meet, so I guess I'll be taking the py2exe approach for now. However, I'm still interested in setting up the associations so that an HTTP content-type header for .egg files could be associated with easy_install. I assume if it were trivial it would have been done by now. http://mail.python.org/pipermail/distutils-sig/2006-September/006734.html has some thoughts from Phillip Eby about automatic execution of easy_install. I will tackle getting my Linux computers to handle an egg/easy_install content-type and then figure out the patches for setuptools so that this can be automatic. Hopefully the Windows situation will be reasonably parallel to Linux. Suggestions and advice are welcome. -- Lloyd Kvam Venix Corp From mhammond at skippinet.com.au Wed Aug 8 16:55:20 2007 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 8 Aug 2007 09:55:20 -0500 Subject: [Distutils] Windows web based installation of multiple packages In-Reply-To: <1186583574.24047.96.camel@localhost.localdomain> References: <1186187389.4695.21.camel@localhost.localdomain><01fb01c7d833$38045c10$6c01a8c0@bobcat> <1186583574.24047.96.camel@localhost.localdomain> Message-ID: <00c301c7d9cc$23cf1c40$6c01a8c0@bobcat> > However, I'm still interested in setting up the associations > so that an HTTP content-type header for .egg files could be > associated with easy_install. I assume if it were trivial it > would have been done by now. But wouldn't this defeat one of the key features of eggs - that you do *not* need to manually locate them and install them? eggs allow you to install a Python program that has external dependencies, and have those external dependencies resolved automatically. If you really do need to manually locate and install a package, you may as well locate and install the bdist_wininst executable, or rpm, or whatever is "native" for the platform in question. I guess if some packages are *only* available as an egg it might make some sense - but I'm yet to get the "egg" religion to the point where I see having an egg as replacing all other distribution formats, so people who do attend that church may beg to differ... Mark From pje at telecommunity.com Wed Aug 8 17:47:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Aug 2007 11:47:10 -0400 Subject: [Distutils] Windows web based installation of multiple packages In-Reply-To: <00c301c7d9cc$23cf1c40$6c01a8c0@bobcat> References: <1186187389.4695.21.camel@localhost.localdomain> <01fb01c7d833$38045c10$6c01a8c0@bobcat> <1186583574.24047.96.camel@localhost.localdomain> <00c301c7d9cc$23cf1c40$6c01a8c0@bobcat> Message-ID: <20070808154445.EA63C3A40A4@sparrow.telecommunity.com> At 09:55 AM 8/8/2007 -0500, Mark Hammond wrote: > > However, I'm still interested in setting up the associations > > so that an HTTP content-type header for .egg files could be > > associated with easy_install. I assume if it were trivial it > > would have been done by now. > >But wouldn't this defeat one of the key features of eggs - that you do *not* >need to manually locate them and install them? Not really. I see the usefulness of this as being able to simply click on a download link in the Cheeseshop after discovering a package's existence (e.g. by reading the Cheeseshop RSS feed). Of course, I personally have a shell window open at all times, so it's no big deal for me to switch over and type "easy_install foo", so defining support for quick-download launching hasn't been a high priority for me. From python at venix.com Wed Aug 8 18:10:26 2007 From: python at venix.com (Lloyd Kvam) Date: Wed, 08 Aug 2007 12:10:26 -0400 Subject: [Distutils] Windows web based installation of multiple packages In-Reply-To: <00c301c7d9cc$23cf1c40$6c01a8c0@bobcat> References: <1186187389.4695.21.camel@localhost.localdomain> <01fb01c7d833$38045c10$6c01a8c0@bobcat> <1186583574.24047.96.camel@localhost.localdomain> <00c301c7d9cc$23cf1c40$6c01a8c0@bobcat> Message-ID: <1186589426.24047.139.camel@localhost.localdomain> On Wed, 2007-08-08 at 09:55 -0500, Mark Hammond wrote: > > However, I'm still interested in setting up the associations > > so that an HTTP content-type header for .egg files could be > > associated with easy_install. I assume if it were trivial it > > would have been done by now. > > But wouldn't this defeat one of the key features of eggs - that you do *not* > need to manually locate them and install them? In my case, someone found my package on a web site and wants to use it. She's not a developer, but simply wants to click and install. My thinking is that an egg describes my package and all of its dependencies. If clicking on the egg ran easy_install, the dependencies would get resolved and missing packages would be installed. > eggs allow you to install a > Python program that has external dependencies, and have those external > dependencies resolved automatically. If you really do need to manually > locate and install a package, you may as well locate and install the > bdist_wininst executable, But then I need to bundle all the dependencies. What if some were already installed? What if I'm inadvertently downgrading packages because I missed a release? > or rpm, or whatever is "native" for the platform yum (or equivalent) would handle the download and installation of the dependencies specified in the rpm. I would not need to bundle additional packages in my rpm. > in question. > > I guess if some packages are *only* available as an egg it might make some > sense - but I'm yet to get the "egg" religion to the point where I see > having an egg as replacing all other distribution formats, so people who do > attend that church may beg to differ... We are probably in agreement about preferring the native packaging. For Linux folks, I'd provide an rpm and expect their packaging system (yum, apt-get, or whatever) to resolve everything. Is there any kind of Windows package manager that would resolve msi dependencies? eggs provide a cross-platform distribution mechanism for Python code. Bypassing the platform package manager is an unfortunate side effect of being cross-platform. People can convert an egg to their preferred package format. -- Lloyd Kvam Venix Corp From miles at jamkit.com Wed Aug 8 19:41:44 2007 From: miles at jamkit.com (Miles) Date: Wed, 08 Aug 2007 18:41:44 +0100 Subject: [Distutils] zc.buildout Message-ID: Hi, We have some a zc.buildout-based deployment system, and we're looking at how we can make the process slightly easier for moving between development, testing and production environments. I was wondering if others have a pattern for setting up a buildout in this way? I am imagining that this is better done by running the different steps, but putting the outputs in different locations (e.g. for testing, put apache configurations in some directory in the test tree), rather than trying to selectively run steps (e.g. for testing, i do not want apache configurations set up). Is there a useful buildout.cfg pattern that covers this? Also, one other use-case that we've got is to be able to group parts together into sets. So, for example, I'd lie to be able to do something like: [buildout] parts = awstats [awstats] parts = awstats-download apacheconf-install crontab-install which would tell buildout to run the three parts listed, rather than remembering all 3 specific parts in the parts list. The advantage being to ensure the bits are always run together. Has anyone done anything like this at all? Thanks, Miles From optilude at gmx.net Wed Aug 8 23:13:02 2007 From: optilude at gmx.net (Martin Aspeli) Date: Wed, 08 Aug 2007 22:13:02 +0100 Subject: [Distutils] zc.buildout In-Reply-To: References: Message-ID: Miles wrote: > Hi, > > We have some a zc.buildout-based deployment system, and we're looking at > how we can make the process slightly easier for moving between > development, testing and production environments. > > I was wondering if others have a pattern for setting up a buildout in > this way? I am imagining that this is better done by running the > different steps, but putting the outputs in different locations (e.g. > for testing, put apache configurations in some directory in the test > tree), rather than trying to selectively run steps (e.g. for testing, i > do not want apache configurations set up). Is there a useful > buildout.cfg pattern that covers this? > > Also, one other use-case that we've got is to be able to group parts > together into sets. So, for example, I'd lie to be able to do something > like: > > [buildout] > parts = awstats > > [awstats] > parts = awstats-download apacheconf-install crontab-install > > which would tell buildout to run the three parts listed, rather than > remembering all 3 specific parts in the parts list. The advantage being > to ensure the bits are always run together. > > Has anyone done anything like this at all? I'm not sure it meets your needs exactly, but you can have one buildout file depend on another. In this case, they are "merged": [buildout] base = basics.cfg ... Here, you get all of basics.cfg, and subsequent parts of the file can override or extend those settings. Sections with the same name are merged as well, so you can override and merge parts of a part, as it were. So, for example, you may have a buildout.cfg that has your normal development setup, and a deployment.cfg that is based on this, but turns off debugging and adds deployment stuff like Apache. Use -c when running ./bin/buildout to specify an alternate file. Martin -- Acquisition is a jealous mistress From zooko at zooko.com Thu Aug 9 02:00:14 2007 From: zooko at zooko.com (zooko) Date: Wed, 8 Aug 2007 18:00:14 -0600 Subject: [Distutils] setuptools controversy (was Re: Prototype setuptools-specific PyPI index.) In-Reply-To: References: <6C905281-8BF2-4FBA-B7E7-8F2F3BEE5EE1@zope.com> <46A23BAE.5090907@v.loewis.de> <932E4411-77F7-42A2-BCFB-B7B5D005EBB6@zope.com> <46A259C4.6090605@v.loewis.de> <20070722164922.AE50D3A40A9@sparrow.telecommunity.com> <9cee7ab80707222156o2bae8a32pdaf7767f8c167918@mail.gmail.com> <20070723053949.EBC3E3A40A9@sparrow.telecommunity.com> <9cee7ab80707231408t24c00125tee04af510724eaa3@mail.gmail.com> Message-ID: <8D398E49-2508-4DD1-802C-5D0AD5C8438D@zooko.com> On Jul 23, 2007, at 3:56 PM, Jim Fulton wrote: > I hope none of the "Zope folks" have said setuptools is evil. We're > big fans and are adopting it whole heartedly. We want it to succeed. > > Speaking for myself, I think it would benefit from being simpler. It > is a lot more complex that I suspect you realize. Of course, there > are things I might do differently. But it is a great accomplishment. > You should be very proud of it and we are very grateful. Yes, hooray for setuptools! I hope that it continues to garner use and support so that it becomes a nexus of cooperation among programmers and users. Regards, Zooko From vedrandekovic at v-programs.com Tue Aug 7 14:13:20 2007 From: vedrandekovic at v-programs.com (veki) Date: Tue, 7 Aug 2007 14:13:20 +0200 Subject: [Distutils] setuptools bbfreeze install egg error Message-ID: <000901c7da7f$dcb946a0$4e01a8c0@ja> Hello, I was download egg file bbfreeze-0.95.2-py2.4-win32.egg ( for windows of course ),and I have setuptools version 0.6c6 after I try install this with setuptools: $ ez_setup bbfreeze-0.95.2-py2.4-win32.egg Traceback (most recent call last): File "C:\Python24\Lib\site-packages\ez_setup.py", line 226, in ? main(sys.argv[1:]) File "C:\Python24\Lib\site-packages\ez_setup.py", line 183, in main from setuptools.command.easy_install import main File "C:\Python24\Lib\site-packages\setuptools\command \easy_install.py", line 16, in ? from distutils import log,dir_util ImportError: cannot import name log Regards, Vedran __________________________________________________________________ Vedran veki ICQ#: 264412055 Current ICQ status: + More ways to contact me Get ICQ! __________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070807/fa323814/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/octet-stream Size: 131 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070807/fa323814/attachment.obj From rmay at ou.edu Thu Aug 9 22:02:45 2007 From: rmay at ou.edu (Ryan May) Date: Thu, 09 Aug 2007 15:02:45 -0500 Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <1609.207.188.248.157.1185477472.squirrel@www.cpcug.org> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> <20070726175840.249E33A403D@sparrow.telecommunity.com> <1609.207.188.248.157.1185477472.squirrel@www.cpcug.org> Message-ID: <46BB72E5.6090903@ou.edu> > On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote: >> At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote: >>> I had the same error. Should I have put in an "import enthought" in all >>> the others? >> No, apparently the manual import doesn't help. Presumably the ones >> you changed need something more like this: >> >> import enthought; enthought.traits = sys.modules['enthought.traits'] >> import enthought.traits; enthought.traits.ui = >> sys.modules['enthought.traits.ui'] >> >> I thought the import would be sufficient, but apparently it's not. :( >> >> Again, let me know if it works so I can change setuptools accordingly. This solution worked for me initially, but now I'm getting warnings when I run any python script I get: warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing __init__.py This is when I added the above lines to the .pth files for both traits and traits.ui. What am I doing wrong? Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From rmay at ou.edu Thu Aug 9 23:09:36 2007 From: rmay at ou.edu (Ryan May) Date: Thu, 09 Aug 2007 16:09:36 -0500 Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <46BB72E5.6090903@ou.edu> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> <20070726175840.249E33A403D@sparrow.telecommunity.com> <1609.207.188.248.157.1185477472.squirrel@www.cpcug.org> <46BB72E5.6090903@ou.edu> Message-ID: <46BB8290.2010200@ou.edu> Ryan May wrote: >> On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote: >>> At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote: >>>> I had the same error. Should I have put in an "import enthought" in all >>>> the others? >>> No, apparently the manual import doesn't help. Presumably the ones >>> you changed need something more like this: >>> >>> import enthought; enthought.traits = sys.modules['enthought.traits'] >>> import enthought.traits; enthought.traits.ui = >>> sys.modules['enthought.traits.ui'] >>> >>> I thought the import would be sufficient, but apparently it's not. :( >>> >>> Again, let me know if it works so I can change setuptools accordingly. > > This solution worked for me initially, but now I'm getting warnings when > I run any python script I get: > > warning: Not importing directory > '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py > warning: Not importing directory > '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing > __init__.py > > > This is when I added the above lines to the .pth files for both traits > and traits.ui. What am I doing wrong? > Turns out that another package (kiva) decided it would be a good idea to put a __init__.py in the enthought directory, which made python angry. Sorry for the noise, distutils. Enthought guys, anybody wanna take a crack at why kiva (and only kiva) insisted on adding a __init__.py to enthought? Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From eucci.group at gmail.com Thu Aug 9 23:10:45 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Thu, 9 Aug 2007 15:10:45 -0600 Subject: [Distutils] Tiny little zc.buildout documentation bug Message-ID: <88d0d31b0708091410w187682d3oc008a84056cd3eba@mail.gmail.com> I'm only on day two or three of our great modernization and buildout experiment. Today I decided to set up my user defaults and try out the download cache. The documentation says: If the file $HOME/.buildout/defaults.cfg, exists, it is read before reading the configuration file So I went ahead and made a "defaults.cfg" in my $HOME/.buildout directory. But it didn't seem to do anything. The problem is that this first mention of the user-defaults config file uses the plural "defaults.cfg". The system itself expects the singular "default.cfg", and the test/example code immediately following the above line uses the singular 'default.cfg'. -- Jeff Shell From dpeterson at enthought.com Thu Aug 9 23:23:00 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 09 Aug 2007 16:23:00 -0500 Subject: [Distutils] [Enthought-dev] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <46BB8290.2010200@ou.edu> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> <20070726175840.249E33A403D@sparrow.telecommunity.com> <1609.207.188.248.157.1185477472.squirrel@www.cpcug.org> <46BB72E5.6090903@ou.edu> <46BB8290.2010200@ou.edu> Message-ID: <46BB85B4.8020206@enthought.com> Ryan May wrote: > Ryan May wrote: > >>> On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote: >>> >>>> At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote: >>>> >>>>> I had the same error. Should I have put in an "import enthought" in all >>>>> the others? >>>>> >>>> No, apparently the manual import doesn't help. Presumably the ones >>>> you changed need something more like this: >>>> >>>> import enthought; enthought.traits = sys.modules['enthought.traits'] >>>> import enthought.traits; enthought.traits.ui = >>>> sys.modules['enthought.traits.ui'] >>>> >>>> I thought the import would be sufficient, but apparently it's not. :( >>>> >>>> Again, let me know if it works so I can change setuptools accordingly. >>>> >> This solution worked for me initially, but now I'm getting warnings when >> I run any python script I get: >> >> warning: Not importing directory >> '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py >> warning: Not importing directory >> '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing >> __init__.py >> >> >> This is when I added the above lines to the .pth files for both traits >> and traits.ui. What am I doing wrong? >> >> > Turns out that another package (kiva) decided it would be a good idea to > put a __init__.py in the enthought directory, which made python angry. > > Sorry for the noise, distutils. > > Enthought guys, anybody wanna take a crack at why kiva (and only kiva) > insisted on adding a __init__.py to enthought? > Well, I just checked the source and 'enthought' is declared as a namespace package in setup.py, and the enthought/__init__.py contains nothing but: try: __import__('pkg_resources').declare_namespace(__name__) except: pass which is what our other projects have. So, since I'm not a bdist_rpm expert, I'm at a loss of even where to begin on this one. -- Dave From sklein at cpcug.org Fri Aug 10 00:52:36 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Thu, 9 Aug 2007 18:52:36 -0400 (EDT) Subject: [Distutils] Namespace packages and egg-info confusion when using bdist_rpm In-Reply-To: <46BB72E5.6090903@ou.edu> References: <64930.207.188.248.157.1185396145.squirrel@www.cpcug.org> <20070725220835.061D23A40BF@sparrow.telecommunity.com> <1537.207.188.248.157.1185467669.squirrel@www.cpcug.org> <20070726164426.C20083A40BF@sparrow.telecommunity.com> <1568.207.188.248.157.1185471477.squirrel@www.cpcug.org> <20070726175840.249E33A403D@sparrow.telecommunity.com> <1609.207.188.248.157.1185477472.squirrel@www.cpcug.org> <46BB72E5.6090903@ou.edu> Message-ID: <11927.207.188.248.157.1186699956.squirrel@www.cpcug.org> On Thu, August 9, 2007 4:02 pm, Ryan May wrote: >> On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote: >>> At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote: >>>> I had the same error. Should I have put in an "import enthought" in >>>> all >>>> the others? >>> No, apparently the manual import doesn't help. Presumably the ones >>> you changed need something more like this: >>> >>> import enthought; enthought.traits = sys.modules['enthought.traits'] >>> import enthought.traits; enthought.traits.ui = >>> sys.modules['enthought.traits.ui'] >>> >>> I thought the import would be sufficient, but apparently it's not. :( >>> >>> Again, let me know if it works so I can change setuptools accordingly. > > This solution worked for me initially, but now I'm getting warnings when > I run any python script I get: > > warning: Not importing directory > '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py > warning: Not importing directory > '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing > __init__.py > > > This is when I added the above lines to the .pth files for both traits > and traits.ui. What am I doing wrong? > > Thanks, > With traits and traits.ui, you have to fix the pth file for all three namespace packages. Most of the others only have one namespace package (enthought). BTW, there is now a totally crazy error I get when trying to do bdist_rpm on kiva. If I do "python setup.py build", it works properly. If I do "python setup.py bdist_rpm" it gives a traceback on the build step that uses (or can be made to use) the same command that worked when run alone, but now complains that it can't fine agg_wrap.c, which doesn't exist (the file is agg_wrap.cpp). Stan Klein From fdrake at acm.org Fri Aug 10 07:42:39 2007 From: fdrake at acm.org (Fred Drake) Date: Fri, 10 Aug 2007 01:42:39 -0400 Subject: [Distutils] Tiny little zc.buildout documentation bug In-Reply-To: <88d0d31b0708091410w187682d3oc008a84056cd3eba@mail.gmail.com> References: <88d0d31b0708091410w187682d3oc008a84056cd3eba@mail.gmail.com> Message-ID: On Aug 9, 2007, at 5:10 PM, Jeff Shell wrote: > The problem is that this first mention of the user-defaults config > file uses the plural "defaults.cfg". The system itself expects the > singular "default.cfg", and the test/example code immediately > following the above line uses the singular 'default.cfg'. Fixed in Subversion. Thanks, Jeff! -Fred -- Fred Drake From sylvain at itaapy.com Fri Aug 10 12:07:13 2007 From: sylvain at itaapy.com (Sylvain Taverne) Date: Fri, 10 Aug 2007 12:07:13 +0200 Subject: [Distutils] Execute Installer in quiet mode Message-ID: <46BC38D1.8020100@itaapy.com> Hi, Is it possible to execute the installer generate with the command : python setup.py bdist_wininst In a quiet mode ? Thank's -- Sylvain From theller at ctypes.org Fri Aug 10 13:37:05 2007 From: theller at ctypes.org (Thomas Heller) Date: Fri, 10 Aug 2007 13:37:05 +0200 Subject: [Distutils] Execute Installer in quiet mode In-Reply-To: <46BC38D1.8020100@itaapy.com> References: <46BC38D1.8020100@itaapy.com> Message-ID: Sylvain Taverne schrieb: > Hi, > Is it possible to execute the installer generate with the command : > > python setup.py bdist_wininst > > In a quiet mode ? > Thank's > No. Thomas From chris at simplistix.co.uk Sun Aug 12 22:59:41 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 12 Aug 2007 21:59:41 +0100 Subject: [Distutils] trouble making egg... Message-ID: <46BF74BD.2000509@simplistix.co.uk> Hi All, I've always built my python packages as a folder, with a __init__.py, any other python bits and then the miscellany (readme.txt, etc) in the same folder. For distributions, I've historically tarred up the folder and put it up to download with installation instructions to just untar somewhere on the python path. However, I'd now like to distribute one of my products as an egg, and so thought I'd use setuptools to build the source distros as well. The package is mailinglogger and you can see the svn trunk here: https://secure.simplistix.co.uk/svn/Simplistix/mailinglogger/trunk/ The setup.py there is wrong in that the egg is builds only contains the 'tests' subpackage and none of the other modules, and no mailinglogger package to boot. Given the component.xml in there, I'm pretty sure I need to add a zip_safe=True, but what else do I need to do to get setup.py to correctly build an egg and a source distro? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Sun Aug 12 23:46:13 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 12 Aug 2007 17:46:13 -0400 Subject: [Distutils] trouble making egg... In-Reply-To: <46BF74BD.2000509@simplistix.co.uk> References: <46BF74BD.2000509@simplistix.co.uk> Message-ID: <20070812214346.69B533A403D@sparrow.telecommunity.com> At 09:59 PM 8/12/2007 +0100, Chris Withers wrote: >Hi All, > >I've always built my python packages as a folder, with a __init__.py, >any other python bits and then the miscellany (readme.txt, etc) in the >same folder. > >For distributions, I've historically tarred up the folder and put it up >to download with installation instructions to just untar somewhere on >the python path. > >However, I'd now like to distribute one of my products as an egg, and so >thought I'd use setuptools to build the source distros as well. > >The package is mailinglogger and you can see the svn trunk here: > >https://secure.simplistix.co.uk/svn/Simplistix/mailinglogger/trunk/ > >The setup.py there is wrong in that the egg is builds only contains the >'tests' subpackage and none of the other modules, and no mailinglogger >package to boot. > >Given the component.xml in there, I'm pretty sure I need to add a >zip_safe=True, but what else do I need to do to get setup.py to >correctly build an egg and a source distro? Short answer: move the package and any data files that should be included to a mailinglogger/ subdirectory. Alternative answer: use package_dir={'mailinglogger':'.'} and manually list the packages (since find_packages() doesn't support package_dir). Downside to this approach is that "setup.py test" and "setup.py develop" will not work correctly -- for you or anyone else who wants to work on your package in source form. From chris at simplistix.co.uk Mon Aug 13 11:46:18 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 13 Aug 2007 10:46:18 +0100 Subject: [Distutils] trouble making egg... In-Reply-To: <20070812214346.69B533A403D@sparrow.telecommunity.com> References: <46BF74BD.2000509@simplistix.co.uk> <20070812214346.69B533A403D@sparrow.telecommunity.com> Message-ID: <46C0286A.3070404@simplistix.co.uk> Phillip J. Eby wrote: >> Given the component.xml in there, I'm pretty sure I need to add a >> zip_safe=True, but what else do I need to do to get setup.py to >> correctly build an egg and a source distro? > > Short answer: move the package and any data files that should be > included to a mailinglogger/ subdirectory. What ends up in the root folder then? just the setup.py? (I want the docs and examples packages in the source distro, I don't know if they make any sense in the egg distro...) > Alternative answer: use package_dir={'mailinglogger':'.'} and manually > list the packages (since find_packages() doesn't support package_dir). I actually tried this before I mailed here, it didn't work but I can't remember the exact reason. I think it may have been that it included the files as modules, but they weren't in a 'mailinglogger' package anymore... > Downside to this approach is that "setup.py test" and "setup.py develop" > will not work correctly -- for you or anyone else who wants to work on > your package in source form. I'm not too fussed about either, tbh ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Mon Aug 13 16:22:50 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 13 Aug 2007 10:22:50 -0400 Subject: [Distutils] Tiny little zc.buildout documentation bug In-Reply-To: <88d0d31b0708091410w187682d3oc008a84056cd3eba@mail.gmail.com> References: <88d0d31b0708091410w187682d3oc008a84056cd3eba@mail.gmail.com> Message-ID: <7E140D40-23AC-46C5-B59B-DFDEA9B81D6F@zope.com> On Aug 9, 2007, at 5:10 PM, Jeff Shell wrote: > I'm only on day two or three of our great modernization and buildout > experiment. Today I decided to set up my user defaults and try out the > download cache. > > The documentation says: > > If the file $HOME/.buildout/defaults.cfg, exists, it is read before > reading the configuration file > > So I went ahead and made a "defaults.cfg" in my $HOME/.buildout > directory. But it didn't seem to do anything. > > The problem is that this first mention of the user-defaults config > file uses the plural "defaults.cfg". The system itself expects the > singular "default.cfg", and the test/example code immediately > following the above line uses the singular 'default.cfg'. Thanks. BTW, you should use launchpad for reporting buildout bugs: https://launchpad.net/zc.buildout/+filebug Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Aug 13 18:17:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Aug 2007 12:17:10 -0400 Subject: [Distutils] trouble making egg... In-Reply-To: <46C0286A.3070404@simplistix.co.uk> References: <46BF74BD.2000509@simplistix.co.uk> <20070812214346.69B533A403D@sparrow.telecommunity.com> <46C0286A.3070404@simplistix.co.uk> Message-ID: <20070813161442.8820A3A40A8@sparrow.telecommunity.com> At 10:46 AM 8/13/2007 +0100, Chris Withers wrote: >Phillip J. Eby wrote: >>>Given the component.xml in there, I'm pretty sure I need to add a >>>zip_safe=True, but what else do I need to do to get setup.py to >>>correctly build an egg and a source distro? >>Short answer: move the package and any data files that should be >>included to a mailinglogger/ subdirectory. > >What ends up in the root folder then? just the setup.py? and anything else you want in the source distro, like the docs and examples. I also usually throw my tests there, and don't distribute them with the egg, but preferences vary. >>Alternative answer: use package_dir={'mailinglogger':'.'} and >>manually list the packages (since find_packages() doesn't support >>package_dir). > >I actually tried this before I mailed here, it didn't work but I >can't remember the exact reason. I think it may have been that it >included the files as modules, but they weren't in a 'mailinglogger' >package anymore... I suspect that you changed package_dir but not packages. You have to explicitly list the packages for that to work. From chris at simplistix.co.uk Tue Aug 14 10:46:17 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 14 Aug 2007 09:46:17 +0100 Subject: [Distutils] trouble making egg... In-Reply-To: <20070813161442.8820A3A40A8@sparrow.telecommunity.com> References: <46BF74BD.2000509@simplistix.co.uk> <20070812214346.69B533A403D@sparrow.telecommunity.com> <46C0286A.3070404@simplistix.co.uk> <20070813161442.8820A3A40A8@sparrow.telecommunity.com> Message-ID: <46C16BD9.2000108@simplistix.co.uk> Phillip J. Eby wrote: > At 10:46 AM 8/13/2007 +0100, Chris Withers wrote: >> Phillip J. Eby wrote: >>>> Given the component.xml in there, I'm pretty sure I need to add a >>>> zip_safe=True, but what else do I need to do to get setup.py to >>>> correctly build an egg and a source distro? >>> Short answer: move the package and any data files that should be >>> included to a mailinglogger/ subdirectory. >> >> What ends up in the root folder then? just the setup.py? > > and anything else you want in the source distro, like the docs and > examples. I also usually throw my tests there, and don't distribute > them with the egg, but preferences vary. Okay, I think I have this now: https://secure.simplistix.co.uk/svn/Simplistix/mailinglogger/tags/3.2.1/ However, when I do: python2.4 setup.py sdist bdist_egg upload ...none of the package metadata gets uploaded: http://pypi.python.org/pypi?:action=display&name=mailinglogger&version=3.2.1 This used to work: http://pypi.python.org/pypi?:action=display&name=mailinglogger&version=3.2.0 ...and I can't see any changes in my setup.py that might be relevent. Any ideas? Is there a problem with PyPi at the moment? >>> Alternative answer: use package_dir={'mailinglogger':'.'} and >>> manually list the packages (since find_packages() doesn't support >>> package_dir). >> >> I actually tried this before I mailed here, it didn't work but I can't >> remember the exact reason. I think it may have been that it included >> the files as modules, but they weren't in a 'mailinglogger' package >> anymore... > > I suspect that you changed package_dir but not packages. You have to > explicitly list the packages for that to work. I tried prettymuch every combination, all with varying degrees of failure... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From sklein at cpcug.org Tue Aug 14 17:48:02 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Tue, 14 Aug 2007 11:48:02 -0400 (EDT) Subject: [Distutils] [Fwd: Re: [Enthought-dev] problem easy_installing kiva: agg.i missing] Message-ID: <13111.207.188.248.157.1187106482.squirrel@www.cpcug.org> FYI - The issues discussed here combine problems specific to the enthought system with problems using setuptools (that are used if available for parts of the packaging for Kiva). ---------------------------- Original Message ---------------------------- Subject: Re: [Enthought-dev] problem easy_installing kiva: agg.i missing From: "Stanley A. Klein" Date: Tue, August 14, 2007 11:46 am To: enthought-dev at mail.enthought.com Cc: "Dave Peterson" -------------------------------------------------------------------------- Dave - I need to remove the files from the Kiva rpm for two reasons. First, they don't belong in python/site-packages on a *nix system if compliance with the *nix File Hierarchy Standard and/or the Linux Standards Baxe (that cites the FHS) is desired. If they are in the rpm they need to go somewhere else. I can fix that by declaring the docs files in the setup.cfg file. I tried that fix, although I don't know if I did it correctly and it didn't resolve both problems. Second, they are causing trouble in running the bdist_rpm because their .pyc and .pyo files are producing an "installed but unpackaged files" error. That is somewhat peculiar to Fedora (and RedHat) because SE-Linux needs to know what files are in sensitive directories or it may think they might have been put there by an intruder and thus block access to them. That problem may be due to the lack of __init__.py files (even blank ones) in the relevant directories. Setuptools apparently looks for __init__.py files to guide its processing. I could put in blank __init__.py files and see what that does. However, I would rather fix both problems (the packaging error and the file placement) at the same time. I figured the easiest way to fix the problem is to get the directories out of Kiva and into somewhere else (a Kiva docs directory that includes tests, examples, and docs). I possibly need to go back, put in the blank __init__.py files in a lot of places (the problem is inconsistent and I can't figure out why that happens) and declare the docs as going into the proper FHS place. Stan Klein On Tue, August 14, 2007 8:58 am, Dave Peterson wrote: > Message: 1 > Date: Mon, 13 Aug 2007 20:23:31 -0500 > From: Dave Peterson > Subject: Re: [Enthought-dev] problem easy_installing kiva: agg.i > missing > > Hi Stan, > > Unfortunately, I've never tried to do a bdist_rpm so I'm at a loss on > what might be going on here. But if I had to guess, I'm wondering if > setuptools uses the .svn dirs for its build process -- these won't be > updated by a local svn mv that hasn't been committed yet so it wouldn't > surprise me that even doing an rm -rf wouldn't get rid of the record of > them. In fact, this seems like it would actively cause the error > mentioned below -- you've removed the file but setuptools thinks they > should exist since there is a record in svn. > > Why do they need to be removed from the rpm? > > > -- Dave > > > Stanley A. Klein wrote: >> I svn-updated my enthought.branches/enthought.kiva. >> >> The build worked. >> >> I'm now back to having problems with unpackaged files and am working on >> that issue, which is becoming very strange as well. >> >> I have tried to get all the files not needed for Kiva operation out into >> a >> separate directory (called kiva_docs) at the enthought.branches level. >> That includes the directories kiva/tests, kiva/agg/tests, >> kiva/agg/examples, and kiva/agg/docs, plus some others that I also can't >> seem to get rid of but don't seem to create the same error condition. I >> did an "svn mv". I went in and did an "rm -rf" of the directories. >> However, it seems that the files I thought I got rid of always seem to >> keep getting found, included as files to be installed, and creating >> problems with "installed but unpackaged files" error of rpm. >> >> >> Stan Klein >> >> >> On Mon, August 13, 2007 2:26 pm, Dave Peterson >> wrote: >> ------------------------------ >> >>> Message: 4 >>> Date: Mon, 13 Aug 2007 12:41:24 -0500 >>> From: Dave Peterson >>> Subject: Re: [Enthought-dev] problem easy_installing kiva: agg.i >>> missing >>> >> >> >>> I'm curious if you've tried this on the latest unstable kiva? The svn >>> version is currently in branches and the source tarball is in the >>> unstable dir. I'm talking about enthought.kiva-2.0.0b2.dev here. >>> >>> -- Dave >>> >>> >>> Stanley A. Klein wrote: >>> >>>> I'm having a very similar problem when I try to do "python setup.py >>>> bdist_rpm" on kiva. When I run "python setup.py build" it completes >>>> properly. The bdist_rpm creates an rpm spec file with the same >>>> command >>>> (or by adding an option I can make it the same command). However, >>>> when >>>> the spec file runs, I get the error message about it not being able to >>>> find agg_wrap.c. >>>> >>>> My traceback (below) points to the error arising in the numpy >>>> distutils. >>>> I wonder if this has anything to do with setuptools being svn aware. >>>> I >>>> can imagine a possible cause being some anomaly in the embedded svn >>>> information for kiva. >>>> >>>> I hope this additional information helps. >>>> >>>> >>>> Stan Klein >>>> >> >> [remainder deleted] -- Stanley A. Klein, D.Sc. Managing Principal Open Secure Energy Control Systems, LLC 8070 Georgia Avenue Silver Spring, MD 20910 301-565-4025 From bjorn at exoweb.net Wed Aug 15 01:15:42 2007 From: bjorn at exoweb.net (=?ISO-8859-1?Q?Bj=F8rn_Stabell?=) Date: Wed, 15 Aug 2007 07:15:42 +0800 Subject: [Distutils] Simpler Python package management: the "egg" command References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> Message-ID: <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> Hi all, [ I originally posted to catalog-sig at python.org, but Jim Fulton recommended I cross-post to this list to as many of the ideas also deal with distutils/setuptools. Please cc replies to catalog- sig at python.org as well. ] I think there's a lot to gain for Python by improving PyPI, and I'm willing to help. I did help a bit with PyPI at last year's EuroPython sprint, and was then made aware of http://wiki.python.org/ moin/CheeseShopDev - is this the most up-to-date plans for PyPI? If you're in a hurry and don't want to read everything; 1) I've created a little app to help prototype how we can do better egg/package management at http://contrib.exoweb.net/trac/browser/egg/ 2) I'd like feedback, and pointers to how I can help more. Basically, the problems I would like to work on solving are: 1) Simplifying/enabling discovery of packages 2) Simplifying/enabling management of packages 3) Improving quality and usefulness of package index From a usability point-of view I'd like to focus on the requirements for the Python newbie, someone that has just discovered Python, but is probably used to package management systems from Linux distributions, FreeBSD, and other dynamic languages like Perl and Ruby (these are also the systems I have experience with, so I'm pulling ideas from them). Ideally everything should be (following Steve Krug's "Don't Make Me Think" recommendations) self-evident, and if that's not possible, at least self-explanatory. Someone put in front of a keyboard without having read any docs should be able to find, install, manage, and perhaps even create Python packages. Better usability will of course benefit everyone, not just beginners. I'm frankly amazed at how people that have programmed Python for years don't really know or use PyPI. I'm convinced making more of Python package system discoverable and easily accessible will greatly improve the adoption of Python, the number of Python packages, and the quality of these packages. I think the typical use cases would be (in order of importance, based on what a typical user would encounter first): * Find available eggs for a particular topic online * Get more information about an egg * Install an egg (and its dependencies) * See which eggs are installed * Upgrade some or all outdated eggs * Remove/uninstall an egg * Create an egg * Find eggs that are plugins for some framework online NAMING So, first of all we'll need either one command, or a set of similarly named commands, to do discovery, installation, and management of packages, as these are common end-user actions. Creation of packages is a bit more advanced, and could be in another command. If there's general agreement that Python eggs is the future way of distributing packages, why not call the command "egg", similar to the way many other package managers are named after the packages, e.g., rpm, port, gem? I'll assume that's the case. Next, where do you find eggs? This might not be a big issue if the "egg" command is configured properly by default, but I'd offer my thoughts. I know the cheeseshop just changed name back to PyPI again. In my opinion, neither of the names are good in that they don't help people remember; any Monty Python connection is lost on the big masses, and PyPI is hard to spell, not very obvious, and a confusing clash with the also-prominent PyPy project. Why not call the place for eggs just eggs? I.e., http://eggs.python.org/ So we'd have the command "egg" for managing eggs that are by default found at "eggs.python.org". I think it's hard to make Python package management more obvious that this. The goal is to get someone that is new to Python to remember how to get and where to find packages, so obvious is a good thing. THE COMMAND LINE PACKAGE MANAGEMENT TOOL The "egg" command should enable you to at least find, show info for, install, and uninstall packages. I think the most common way to do command line tools like this is to offer sub-commands, a la, bzr, port, svn, apt-get, gem, so I suggest: egg - list out a help of commands egg search - search for eggs (aliases: find/list) egg info - show info for egg (aliases: show/details) egg install - install named eggs egg uninstall - uninstall eggs (aliases: remove/purge/delete) so you can do: egg search bittorrent to find all packages that have anything to do with bittorrent (full- text search of the package index), and then: egg install iTorrent to actually download and install the package. PROTOTYPE I've built a prototype for a command that works this way, implementing most (except the last) of the use cases at least partiall. You can give it a go as follows: # install prerequisities on your platform # e.g., sudo apt-get install python-setuptools sqlite3 libsqlite3-0 python-pysqlite2 svn co http://contrib.exoweb.net/svn/egg/ cd egg sudo python setup.py develop # should install storm for you gzip -dc pypi.sql.gz | sqlite3 ~/.pythoneggs.db # bootstrap cache egg sync # update cache It's still incomplete, lacking tests, might only work on unix-y computers, and is lacking support for lots of features like activation/deactivation, and upgrades, but it works for basic stuff like finding, installing, and uninstalling packages. Summary of the design: * Local and PyPI package information is synchronized into a local sqlite database for easy access * Storm is used for ORM (but could easily be changed) * Installation is handled by passing off the "egg install" command to "easy_install" * I'm using a non-standard command-line parser (but could easily be changed) * For interactive use on terminals that supports it: colorizes and adjusts text to fit While doing the synchronization with PyPI I discovered a couple of issues, described below, that makes the application unfit for common use yet. (Eg., it has to query the PyPI for each of the packages.) Most subcommands take arguments that can be a free mix of set names and query strings. I thought this would make for the most forgiving and user-friendly interface. These are filters; by default all eggs match. SETS: Eggs have a few attributes that can be used to limit to a subset of all eggs, e.g., whether it is installed, active, oudated, local, or remote. Specifying several of these creates a join of the sets, it further limits the number of eggs. QUERY STRINGS: If none of the set names are matched, the argument is assumed to be a query string. Many subcommands like "search" do a full-text search of the package cache database. Others, like "list", will do a substring match of package names. Others, like "install" will require you to match the name exactly. You can specify a specific version by adding a slash, e.g., "name/version". Here are some example commands: egg list installed sql - list all installed eggs having sql in their name egg search installed sql - list all installed eggs mentioning sql anywhere in the package metadata egg list oudated installed - list all outdated installed eggs egg list oudated active - list all outdated and active (and installed) eggs egg uninstall outdated - uninstall all oudated eggs egg info pysqlite - show information about pysqlite egg info pysqlite/2.0.0 - show information about version 2.0.0 of pysqlite egg sync local - rescan local packages and update cache db PYPI IMPROVEMENT SUGGESTIONS While doing the application I discovered one important missing feature: PyPI doesn't offer a way to programatically bulk-download information about all eggs, as is customary for many other packaging systems. This means "egg sync" will have to fetch the information for each package individually. I think it wouldn't be hard to offer a compressed XML file with all of the package information, suitable for download. A minor nuiscence is that there's no way to get only eggs/ distributions; PyPI lists packages, and some packages don't even have any eggs. The "egg" command will try to download each of these empty packages at each sync (since it treats empty packages as "packages for which we haven't downloaded eggs for yet"). It might be better to list eggs/distributions instead of packages. There's a lot of opportunity in improving the consistency and usefulness of package metainformation. Once you have it all sync'ed to a local SQlite database and start snooping around, it'll be pretty obvious; very few packages use the dependencies etc. (In fact, I think the dependencies/obsoletes definitions are overengineered; we could get by with just a simple package >= version number). Many people use other platform-specific packaging system to manage Python packages, probably both because this gives dependencies to other non-Python packages, but also because PyPI hasn't been very useful or easy to use. It may even be asked what the role of PyPI is since it's never going to replace platform-specific packaging systems; then should it support them? How? In any case, installing Python packages from different packaging systems would result in problems, and currently "egg" can't find Python packages installed using other systems. ("Yolk" has some support for discovering Python packages installed using Gentoo.) Optional: These days XMLRPC (and the WS-Deathstar) seems to be losing steam to REST, so I think we'd gain a lot of "hackability" by enabling a REST interface for accessing packages. Eventually we probably need to enforce package signing. EGG IDEAS It'd be good for "egg" to support both system- and user-wide configurations, and to support downloading from several package indexes, like apt-get does. Perhaps "egg" should keep the uninstalled packages in a cache, like apt-get and I believe buildout. Perhaps "egg" should provide a simple web server to allow browsing (and perhaps installation from) local packages (I believe the Ruby guys have this). If this web server should be discoverable via Bonjour/Zeroconf, then all that's needed to set up a cache of PyPI is to run an egg server (that people on the net auto-discovers) and regularly download all packages. How could "egg" work with "buildout"? Should buildout be used for project-specific egg installations? There area lot of other ideas and thoughts in the TODO.txt file of the egg command, and I agree it's a good idea to join forces (distutils, setuptools, yolk, enstaller, PythonEggTools, ...). Rgds, Bjorn From bjorn at exoweb.net Wed Aug 15 02:15:48 2007 From: bjorn at exoweb.net (=?ISO-8859-1?Q?Bj=F8rn_Stabell?=) Date: Wed, 15 Aug 2007 08:15:48 +0800 Subject: [Distutils] PyPI - Evolve our own or reuse existing package systems? In-Reply-To: <200708150037.57652.paul@boddie.org.uk> References: <200708150037.57652.paul@boddie.org.uk> Message-ID: <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> (Since my email was a bit long and wide I'm trying to update the subject when the response is rather focused.) On Aug 15, 2007, at 06:37, Paul Boddie wrote: > Bj?rn Stabell wrote: [...] > I've been moderately negative about evolving a parallel > infrastructure to > other package and dependency management systems in the past, and > I'm not > enthusiastic about things like CPAN or language-specific > equivalents. The > first thing most people using a GNU/Linux or *BSD distribution are > likely to > wonder is, "Where are the Python packages in my package selector?" > > There are exceptions, of course. Some people may be sufficiently > indoctrinated > in the ways of Python, which I doubt is the case for a lot of > people looking > for packages. Others may be working in restricted environments > where system > package management tools don't really help. And people coming from > Perl might > wonder where the CPAN equivalent is, but they should also remind > themselves > what the system provides - they have manpages for Perl, after all. [...] > I've read through the text that I've mercilessly cut from this > response, and I > admire the scope of this effort, but I do wonder whether we > couldn't make use > of existing projects (as others have noted), and not only at the > Python-specific level, especially since the user interface to the > "egg" tool > seems to strongly resemble other established tools - as you seem to > admit in > this and later messages, Bj?rn. [...] > I was thinking of re-using the Debian indexing strategy. It's very > simple, > perhaps almost quaintly so, but a lot of the problems revealed with > the > current strategies around PyPI (not exactly mitigated by bizarre > tool-related > constraints) could be solved by adopting existing well-worn > techniques. [...] > If I recall correctly, the PEP concerned just "bailed" on the version > numbering and dependency management issue, despite seeming to be > inspired by > Debian or RPM-style syntax. [...] > As I've said before, it's arguably best to work with whatever is > already > there, particularly because of the "interface" issue you mention with > non-Python packages. I suppose the apparent lack of an open and > widespread > package/dependency management system on Windows (and some UNIX > flavours) can > be used as a justification to write something entirely new, but I > imagine > that only very specific tools need writing in order to make existing > distribution mechanisms work with Windows - there's no need to > duplicate > existing work from end to end "just because". [...] > Agreed. And by adopting existing mechanisms, we can hopefully avoid > having to > reinvent their feature sets, too. > > P.S. Sorry if this sounds a bit negative, but I've been reading the > archives > of the catalog-sig for a while now, and it's a bit painful reading > about how > sensitive various projects are to downtime in PyPI, how various > workarounds > have been devised with accompanying whisper campaigns to tell > people where > unofficial mirrors are, all whilst the business of package > distribution > continues uninterrupted in numerous other communities. > > If I had a critical need to get Python packages directly from their > authors to > run on a Windows machine, for example, I'd want to know how to do > so via a > Debian package channel or something like that. This isn't original > thought: > I'm sure that Ximian Red Carpet and Red Hat Network address many > related > issues. There seems to be two issues: 1) Should Python have its own package management system (with dependencies etc) in parallel with what's already on many platforms (at least Linux and OS X)? Anyone that has worked with two parallel package management systems knows that dependencies are hellish. * If you mix and match you often end up with two of everything. * It'll be incomplete because you can't easily specify dependencies to non-Python packages. 2) If we agree Python should have a package management system, should we build or repurpose some other one? * I think it's a matter of pride and proof of concept to have one written in Python. That doesn't mean we can't get ideas from others. * It's also not that hard to do. The prototype I threw up took one weekend + half a day, and consists of about 500 lines of new code. It could be refactored and made smaller, but even if a complete version is ten times the size of that, it's still not a huge undertaking. * With a Python version we could relatively easily innovate beyond what traditional packaging systems do; ports and apt are pretty much stagnated. I think RubyGems seems to have some cool features, features that probably wouldn't have happened if they were using ports or apt-get (but then they could piggyback on innovations in those tools, I guess). If it works for them, why shouldn't it work for us? * It would have to be as portable as Python is; many packaging systems are by nature relatively platform-specific. * If we don't build our own, doesn't that mean we throw out eggs? * Packaging systems are useful for mega frameworks like Zope, TurboGears, and Django, and slightly less so for projects you roll on your own, to manage distribution and installation of plugins and addons. Relying on platform-specific packaging systems for these may not work that well. (But I could be wrong about that.) That said, it might be possible to do some kind of hybrid, for PyPI to be a "meta package" repository that can easily feed into platform specific packaging systems. And to perhaps also have a client-side "meta package manager" that will call upon the platform-specific package manager to install stuff. It looks like, for example, ports have targets to build to other systems, e.g., pkg, mpkg, dmg, rpm, srpm, dpkg. So maintaining package information in (or compatible with) ports could make it easy to feed packages into other package systems. * Benefit: We're working with other package systems, just making it easier to get Python packages into them. * Drawback: They may not want to include all packages, at the speed at which we want, or the way we want to. (I.e., there may still be packages you'd want that are only available on PyPI.) * Drawback: Some systems don't have package systems. Which brings me to: If we're just distributing source files why don't we use a source control system such as svn, bzr, or hg? The package developers have trunk, PyPI is a branch, the platform-specific package maintainers have a branch, and what's installed onto your system is in the end a branch (serially connected). Some systems, like Subversion, can also include externals like I did with cliutils on the egg package. Just a thought. Rgds, Bjorn From arve.knudsen at gmail.com Wed Aug 15 15:34:41 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Wed, 15 Aug 2007 15:34:41 +0200 Subject: [Distutils] PyPI - Evolve our own or reuse existing package systems? In-Reply-To: <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> References: <200708150037.57652.paul@boddie.org.uk> <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> Message-ID: Hei Bj?rn :) These are some interesting points you are making. I have in fact been developing a general software deployment system, Conduit, in Python for some time, that is capable of supporting several major platforms (at the moment: Linux, Windows and OS X). It's not reached any widespread use, but we (at the Simula Research Laboratory) are using it to distribute software to students attending some courses at the university of Oslo. Right now we are in the middle of preparing for the semester start, which is next week. The system is designed to be general, both with regard to the target platform and the deployable software. I've solved the distribution problem by constructing an XML-RPC Web service, that serves information about software projects in RDF (based on the DOAP format). This distribution service is general and independent of the installation system, which acts as a client of the latter. If this sounds interesting to you I'd love if you checked it out and gave me some feedback. It is an experimental project, and as such we are definitely interested in ideas/help from others. Arve On 8/15/07, Bj?rn Stabell wrote: > > (Since my email was a bit long and wide I'm trying to update the > subject when the response is rather focused.) > > On Aug 15, 2007, at 06:37, Paul Boddie wrote: > > Bj?rn Stabell wrote: > [...] > > I've been moderately negative about evolving a parallel > > infrastructure to > > other package and dependency management systems in the past, and > > I'm not > > enthusiastic about things like CPAN or language-specific > > equivalents. The > > first thing most people using a GNU/Linux or *BSD distribution are > > likely to > > wonder is, "Where are the Python packages in my package selector?" > > > > There are exceptions, of course. Some people may be sufficiently > > indoctrinated > > in the ways of Python, which I doubt is the case for a lot of > > people looking > > for packages. Others may be working in restricted environments > > where system > > package management tools don't really help. And people coming from > > Perl might > > wonder where the CPAN equivalent is, but they should also remind > > themselves > > what the system provides - they have manpages for Perl, after all. > [...] > > I've read through the text that I've mercilessly cut from this > > response, and I > > admire the scope of this effort, but I do wonder whether we > > couldn't make use > > of existing projects (as others have noted), and not only at the > > Python-specific level, especially since the user interface to the > > "egg" tool > > seems to strongly resemble other established tools - as you seem to > > admit in > > this and later messages, Bj?rn. > [...] > > I was thinking of re-using the Debian indexing strategy. It's very > > simple, > > perhaps almost quaintly so, but a lot of the problems revealed with > > the > > current strategies around PyPI (not exactly mitigated by bizarre > > tool-related > > constraints) could be solved by adopting existing well-worn > > techniques. > [...] > > If I recall correctly, the PEP concerned just "bailed" on the version > > numbering and dependency management issue, despite seeming to be > > inspired by > > Debian or RPM-style syntax. > [...] > > As I've said before, it's arguably best to work with whatever is > > already > > there, particularly because of the "interface" issue you mention with > > non-Python packages. I suppose the apparent lack of an open and > > widespread > > package/dependency management system on Windows (and some UNIX > > flavours) can > > be used as a justification to write something entirely new, but I > > imagine > > that only very specific tools need writing in order to make existing > > distribution mechanisms work with Windows - there's no need to > > duplicate > > existing work from end to end "just because". > [...] > > Agreed. And by adopting existing mechanisms, we can hopefully avoid > > having to > > reinvent their feature sets, too. > > > > P.S. Sorry if this sounds a bit negative, but I've been reading the > > archives > > of the catalog-sig for a while now, and it's a bit painful reading > > about how > > sensitive various projects are to downtime in PyPI, how various > > workarounds > > have been devised with accompanying whisper campaigns to tell > > people where > > unofficial mirrors are, all whilst the business of package > > distribution > > continues uninterrupted in numerous other communities. > > > > If I had a critical need to get Python packages directly from their > > authors to > > run on a Windows machine, for example, I'd want to know how to do > > so via a > > Debian package channel or something like that. This isn't original > > thought: > > I'm sure that Ximian Red Carpet and Red Hat Network address many > > related > > issues. > > There seems to be two issues: > > 1) Should Python have its own package management system (with > dependencies etc) in parallel with what's already on many platforms > (at least Linux and OS X)? Anyone that has worked with two parallel > package management systems knows that dependencies are hellish. > > * If you mix and match you often end up with two of everything. > > * It'll be incomplete because you can't easily specify > dependencies to non-Python packages. > > > 2) If we agree Python should have a package management system, should > we build or repurpose some other one? > > * I think it's a matter of pride and proof of concept to have one > written in Python. That doesn't mean we can't get ideas from others. > > * It's also not that hard to do. The prototype I threw up took > one weekend + half a day, and consists of about 500 lines of new > code. It could be refactored and made smaller, but even if a > complete version is ten times the size of that, it's still not a huge > undertaking. > > * With a Python version we could relatively easily innovate beyond > what traditional packaging systems do; ports and apt are pretty much > stagnated. I think RubyGems seems to have some cool features, > features that probably wouldn't have happened if they were using > ports or apt-get (but then they could piggyback on innovations in > those tools, I guess). If it works for them, why shouldn't it work > for us? > > * It would have to be as portable as Python is; many packaging > systems are by nature relatively platform-specific. > > * If we don't build our own, doesn't that mean we throw out eggs? > > * Packaging systems are useful for mega frameworks like Zope, > TurboGears, and Django, and slightly less so for projects you roll on > your own, to manage distribution and installation of plugins and > addons. Relying on platform-specific packaging systems for these may > not work that well. (But I could be wrong about that.) > > > That said, it might be possible to do some kind of hybrid, for PyPI > to be a "meta package" repository that can easily feed into platform > specific packaging systems. And to perhaps also have a client-side > "meta package manager" that will call upon the platform-specific > package manager to install stuff. > > It looks like, for example, ports have targets to build to other > systems, e.g., pkg, mpkg, dmg, rpm, srpm, dpkg. So maintaining > package information in (or compatible with) ports could make it easy > to feed packages into other package systems. > > * Benefit: We're working with other package systems, just making > it easier to get Python packages into them. > > * Drawback: They may not want to include all packages, at the > speed at which we want, or the way we want to. (I.e., there may > still be packages you'd want that are only available on PyPI.) > > * Drawback: Some systems don't have package systems. > > > Which brings me to: If we're just distributing source files why don't > we use a source control system such as svn, bzr, or hg? The package > developers have trunk, PyPI is a branch, the platform-specific > package maintainers have a branch, and what's installed onto your > system is in the end a branch (serially connected). Some systems, > like Subversion, can also include externals like I did with cliutils > on the egg package. Just a thought. > > > Rgds, > Bjorn > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070815/c33a2a13/attachment.htm From bjorn at exoweb.net Thu Aug 16 10:28:12 2007 From: bjorn at exoweb.net (=?ISO-8859-1?Q?Bj=F8rn_Stabell?=) Date: Thu, 16 Aug 2007 16:28:12 +0800 Subject: [Distutils] [Catalog-sig] PyPI - Evolve our own or reuse existing package systems? In-Reply-To: References: <200708150037.57652.paul@boddie.org.uk> <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> Message-ID: <865235D6-2674-4B78-ABE8-4DFC5919576E@exoweb.net> On Aug 15, 2007, at 21:34, Arve Knudsen wrote: > These are some interesting points you are making. I have in fact > been developing a general software deployment system, Conduit, in > Python for some time, that is capable of supporting several major > platforms (at the moment: Linux, Windows and OS X). It's not > reached any widespread use, but we (at the Simula Research > Laboratory) are using it to distribute software to students > attending some courses at the university of Oslo. Right now we are > in the middle of preparing for the semester start, which is next week. > > The system is designed to be general, both with regard to the > target platform and the deployable software. I've solved the > distribution problem by constructing an XML-RPC Web service, that > serves information about software projects in RDF (based on the > DOAP format). This distribution service is general and independent > of the installation system, which acts as a client of the latter. > > If this sounds interesting to you I'd love if you checked it out > and gave me some feedback. It is an experimental project, and as > such we are definitely interested in ideas/help from others. Hi Arve, That's an interesting coincidence! :) Without turning it into a big research project, it would be interesting to hear what you (honestly) thought were the strengths and weaknesses of Conduit compared to, say, deb/rpm/ports/emerge, whichever you have experience with. I did download and look at Conduit, but haven't tried it yet. There are so many ways to take this, and so many "strategic" decisions that I'd hope people on the list could help out with. Personally I think it would be great if we had a strong Python-based central package system, perhaps based on Conduit. I'm pretty sure Conduit would have to have the client and server-side components even more clearly separated, though, and the interface between them open and clearly defined (which I think it is, but it would have to be discussed). I see Conduit (and PyPI) supports DOAP, and looking around I also found http://python.codezoo.com/ by O'Reilly; it also seems to have a few good ideas, for example voting and some quality control (although that's a very difficult decision, I guess). Rgds, Bjorn From eu at lbruno.org Thu Aug 16 19:12:12 2007 From: eu at lbruno.org (=?UTF-8?Q?Lu=C3=ADs_Bruno?=) Date: Thu, 16 Aug 2007 18:12:12 +0100 Subject: [Distutils] Simpler Python package management: the "egg" command In-Reply-To: <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> Message-ID: <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> 'lo there! Bj?rn Stabell: > * Find available eggs for a particular topic online > * Get more information about an egg > * Install an egg (and its dependencies) > * See which eggs are installed > * Upgrade some or all outdated eggs > * Remove/uninstall an egg > * Create an egg > * Find eggs that are plugins for some framework online Having a checklist of use cases is useful, as others can add to it (or shoot items down). Thanks. > egg - list out a help of commands > egg search - search for eggs (aliases: find/list) > egg info - show info for egg (aliases: show/details) > egg install - install named eggs > egg uninstall - uninstall eggs (aliases: remove/purge/delete) > [...] > egg list installed sql - list all installed eggs having sql in their name > egg search installed sql - list all installed eggs mentioning sql anywhere [...] > egg list oudated installed - list all outdated installed eggs > egg list oudated active - list all outdated and active (and installed) eggs > egg uninstall outdated - uninstall all oudated eggs > egg info pysqlite - show information about pysqlite > egg info pysqlite/2.0.0 - show information about version 2.0.0 of pysqlite > egg sync local - rescan local packages and update cache db Sorry, but I think you meant apt-get instead of egg. No, I didn't search the archives. But making an apt-get repository (yum, emerge...) can't be *that* hard; it also can't be an uncommon idea. Someone must have suggested it before. On second thought, if I recall correctly Debian-style repositories have to update a master Packages catalog for *each* and *every* *single* upload. That's a -1. I think you've asked for a "sync local" master and I snipped it. Any other -1? We'll, getting the repositores updated for each single upload becomes O(N), but it's a small N anyway. One per supported repository format. > Optional: These days XMLRPC (and the WS-Deathstar) seems to be losing > steam to REST, so I think we'd gain a lot of "hackability" by > enabling a REST interface for accessing packages. Yep. Me likes dispatching on Accept: to get different responses. I think Apache can do it with a type-map. Gotta read up on the performance of it. That was an idea I stumbled upon during the recent PyPI discussions. Well, me likes flat files. > Eventually we probably need to enforce package signing. Heh, like the .deb and .rpm signatures? As I've said previously, I'd like to have a standard-type-repository for PyPI. If we're distributing binaries (has Phillip Eby said, sdist works *fine* for source tarballs ) there are already people working on that subject. Package signing's one of the for-free wheels we don't have to invent. Squared wheels and all that. So digital signatures' a +1. > EGG IDEAS > > [snip] > Perhaps "egg" should provide a simple web server to allow browsing > (and perhaps installation from) local packages. D*mn. Right now you just serve your .../site-packages and you can easy_install from it (I think Phillip Eby said as much recently). This standard-type-repository idea makes that a tad more difficult. > If this web server should be discoverable via > Bonjour/Zeroconf, then all that's needed to set up a cache of PyPI is > to run an egg server (that people on the net auto-discovers) and > regularly download all packages. Maybe regenerating a bunch of static files isn't that difficult anyway; do it before serving content. Well, you're gonna run a local PyPI copy; might as well run the PyPI code anyway. And now the collective asks: who the fsck is this Luis Bruno idiot? Just one more Python user with some free time on his hands. Greetings, -- Luis Bruno From dpeterson at enthought.com Thu Aug 16 19:30:46 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 16 Aug 2007 12:30:46 -0500 Subject: [Distutils] Where is easy_install finding this dependency from? Message-ID: <46C489C6.6070300@enthought.com> We've gotten a number of problem reports from people trying to install our egg releases where there doesn't seem to be enough information being output to the console to really debug the problem. Here's an illustrating log taken from a user's report: ======= C:\Documents and Settings\Gary>easy_install -f http://code.enthought.com/enstaller/eggs/windows/xp ets Searching for ets Reading http://code.enthought.com/enstaller/eggs/windows/xp Best match: ets 2.5b2 Downloading http://code.enthought.com/enstaller/eggs/windows/xp/ets-2.5b2-py2.5.egg Processing ets-2.5b2-py2.5.egg Moving ets-2.5b2-py2.5.egg to c:\python25\lib\site-packages Adding ets 2.5b2 to easy-install.pth file Installed c:\python25\lib\site-packages\ets-2.5b2-py2.5.egg Reading http://code.enthought.com/enstaller/eggs/source Reading http://code.enthought.com/enstaller/eggs/source/unstable Processing dependencies for ets Searching for wx==2.6 Reading http://cheeseshop.python.org/pypi/wx/ Couldn't find index page for 'wx' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://cheeseshop.python.org/pypi/ No local packages or download links found for wx==2.6 error: Could not find suitable distribution for Requirement.parse ('wx==2.6') ======= I've verified that our 'ets' egg does not even mention 'wx' in its setup.py so something else on this user's system must be requiring it, right? But there's no information here about where the requirement came from. In fact the situation is even worse than that because easy_install outputs the line about processing dependencies for ets, but the next step is about some other egg's dependencies and not one of the 20+ dependencies actually listed in the 'ets' egg. Can easy_install be updated to inform the user of which egg, or the first one if there are many of them, that generated the requirement on each thing it is trying to install? I'm thinking of a change in the line that looks like: Searching for wx==2.6 to something like Searching for wx==2.6 (required by foo.bar-1.0) It also seems like the error line should indicate which egg generated the requirement for 'wx==2.6', right? -- Dave From eucci.group at gmail.com Thu Aug 16 20:20:21 2007 From: eucci.group at gmail.com (Jeff Shell) Date: Thu, 16 Aug 2007 12:20:21 -0600 Subject: [Distutils] [Catalog-sig] PyPI - Evolve our own or reuse existing package systems? In-Reply-To: <865235D6-2674-4B78-ABE8-4DFC5919576E@exoweb.net> References: <200708150037.57652.paul@boddie.org.uk> <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> <865235D6-2674-4B78-ABE8-4DFC5919576E@exoweb.net> Message-ID: <88d0d31b0708161120u54fc1d38ib50366d2125acc7c@mail.gmail.com> On 8/14/07, Bj?rn Stabell wrote: > There seems to be two issues: > > 1) Should Python have its own package management system (with > dependencies etc) in parallel with what's already on many platforms > (at least Linux and OS X)? Anyone that has worked with two parallel > package management systems knows that dependencies are hellish. > > * If you mix and match you often end up with two of everything. > > * It'll be incomplete because you can't easily specify > dependencies to non-Python packages. On that second bullet, tools like 'buildout' seem better equipped for handling those situations. Yesterday I saw a `buildout.cfg` for building and testing `lxml` against fresh downloads and builds of libxml2 and libxslt. It downloaded and built those two things locally before getting, building, and installing the `lxml` egg locally. Platform package management terrifies me. I work in Python far more than I work in a particular operating system (even though our office is pretty much Mac OS X and FreeBSD based). It's very easy for our servers to get stuck at a particular FreeBSD version, while the ports move on. Eventually they get so out of date that ports are pretty much unusable. > 2) If we agree Python should have a package management system, should > we build or repurpose some other one? > > [snip] > > * With a Python version we could relatively easily innovate beyond > what traditional packaging systems do; ports and apt are pretty much > stagnated. I think RubyGems seems to have some cool features, > features that probably wouldn't have happened if they were using > ports or apt-get (but then they could piggyback on innovations in > those tools, I guess). If it works for them, why shouldn't it work > for us? I agree. > * It would have to be as portable as Python is; many packaging > systems are by nature relatively platform-specific. You could change "have to" to "gets to" there. :). This is a big plus -- I know how easy_install and `gem` work as I use them far more frequently on both my desktop and various servers than any platform-specific packaging system. > * Packaging systems are useful for mega frameworks like Zope, > TurboGears, and Django, and slightly less so for projects you roll on > your own, to manage distribution and installation of plugins and > addons. Relying on platform-specific packaging systems for these may > not work that well. (But I could be wrong about that.) Personally, I think packaging systems are worse here. But I just may be a control freak... And I've had the luxury of Zope being a big self contained package for quite some time. Now that it's breaking into smaller pieces, it gets a bit more complex, but the combination of `setuptools` and `buildout` seem to be doing their jobs admirably. Relatively admirably. Once you have Ruby and Gems, Ruby on Rails installs with just one line:: gem install rails --include-dependencies I think Pylons and/or Turbogears does just about the same..? It's been a while since I looked at either of them. But that one line is a lot easier to work with than:: If running Debian, run ``apt-get ....`` If running RedHat or RPM system, .... If running Mac OS X with MacPorts, run ... If running ... then ... > That said, it might be possible to do some kind of hybrid, for PyPI > to be a "meta package" repository that can easily feed into platform > specific packaging systems. And to perhaps also have a client-side > "meta package manager" that will call upon the platform-specific > package manager to install stuff. For my own experience, that sounds worse. However, it would be nice if 'egg' could detect that certain things were installed by a non-egg system (ie, having `py-sqlite` from MacPorts) and not installing it. This goes into a deeper frustration I've had in the past: I installed MySQL on my desktop (Mac OS X) using a disk image / .pkg installer downloaded from MySQL's web site. Then I think I tried installing a python package from MacPorts (maybe just the mysql bindings?) that had a MySQL dependency. It didn't detect that I already had MySQL installed, and MacPorts then tried installing it on its own. At that point, I stopped using ports for just about anything Python related, aside from getting Python and Py-Readline. It was easier to use easy_install or regular distutils and the like. The dependencies were met, but not advertised in a way that was friendly to the packaging system in question. > It looks like, for example, ports have targets to build to other > systems, e.g., pkg, mpkg, dmg, rpm, srpm, dpkg. So maintaining > package information in (or compatible with) ports could make it easy > to feed packages into other package systems. > > * Benefit: We're working with other package systems, just making > it easier to get Python packages into them. > > * Drawback: They may not want to include all packages, at the > speed at which we want, or the way we want to. (I.e., there may > still be packages you'd want that are only available on PyPI.) Or packages you only want internally. Or packages you don't want available on PyPI because they're very specific to a large framework/toolkit like Zope 3. > * Drawback: Some systems don't have package systems. And some administrators don't use them beyond (maybe) initially setting up the system. I also don't know how well those package systems deal with concepts like local-installs. Not just local to a single user, but local to a single package. `zc.buildout` is good about this, almost to a fault. There is a rough balance there between desktop and personal machine global-install ease of use and being able to set up fine-tuned self-contained setups. Anyways, I'd vote pure-python. Even on the most barren of machines, it's relatively easy to build and install Python from source. Even on a fairly old installation, it's easy to build an install a new version of Python from source - probably far easier than wrestling with the package manager about updating its database and then updating package after package after package after package that one doesn't want. I think that Python should be all that you need in order to get other Python packages. `easy_install` pretty much gives us this today. There are improvements I'd love to see - reports of what I have installed, what's active, what's taking precedence in my environment, etc. Your tool may do this, I haven't had time to look yet. Ruby's 'gem' command does this beautifully. And I hardly ever touch Ruby or gems; it was just very easy to use for the few things I've wanted to try. -- Jeff Shell From dpeterson at enthought.com Thu Aug 16 20:38:43 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 16 Aug 2007 13:38:43 -0500 Subject: [Distutils] Finding eggs when dependency_links and user specified URLs both given to easy_install? Message-ID: <46C499B3.8020500@enthought.com> What happens if a user specifies a URL via a '-f' option on the easy_install command line but our published eggs specify dependency_links in the setup.py? Will easy_install search the user's specified location first, last, or will it not search them at all? (I'm hoping for the first!) I'm asking because we want to provide binaries (eggs) of our projects for various platforms (RHEL3, RHEL4, RHEL5, Ubuntu 7.04, etc.) all of which generate the same egg name, but our projects include C /C++ extensions that require compilation for the specific platform, so we've structured an egg repository that allows us to separate these equivalently named eggs by the actual target platform. We also want users to be able to install from source tar/zip balls for platforms we haven't built binaries for yet, so our eggs all include dependency_links to point at our egg repo. Due to issues with build management, we aren't currently listing the target platform URLs in each egg's setup.py's dependency_links when it gets built on that target platform. It just has the source tar/zip ball repository's URL. But early indications are that when a user installs by doing something like: easy_install -f ets==2.5b2 then the first egg gets found properly in the platform specific repo, but all of its dependencies are only being searched for in the source repo. Is this just a misinterpretation of the output generated by easy_install? BTW, it would be nice if there was an option to name your generated eggs based on a full specification of OS and version, hardware arch, etc. And of course, the whole setuptools universe of tools would need to know to search for eggs with that level of accuracy before dropping down to more generic specs. Another similar use case: Some of our users want to locally cache eggs for distribution within their corporate environment for various reasons (eggs tested and approved to work with their corporate desktop standard, etc.) Those users would also be specifying a '-f' option to easy_install to point at their corporate cache. However, all of our eggs will have dependency_links specified. Will these corporate users be able to install and pull things from their local repository, or will easy_install always look at our dependency_links' specified locations first? It doesn't seem convenient to force them to have to remember to provide a '-H' setting, though I guess their IT group could create an alias or batch script that does this for them. I also guess that their IT group could generate their own version of easy_install that customizes / hard-codes the URLs that it would look at, but that seems like kind of a long way to go to solve this problem. -- Dave From barry at python.org Thu Aug 16 21:46:56 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Aug 2007 15:46:56 -0400 Subject: [Distutils] Simpler Python package management: the "egg" command In-Reply-To: <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> Message-ID: <1D4917BC-BD94-44CE-BFEC-64F385B69561@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 14, 2007, at 7:15 PM, Bj?rn Stabell wrote: > If there's > general agreement that Python eggs is the future way of distributing > packages, why not call the command "egg", similar to the way many > other package managers are named after the packages, e.g., rpm, port, > gem? +1 > Next, where do you find eggs? This might not be a big issue if the > "egg" command is configured properly by default, but I'd offer my > thoughts. I know the cheeseshop just changed name back to PyPI > again. In my opinion, neither of the names are good in that they > don't help people remember; any Monty Python connection is lost on > the big masses, and PyPI is hard to spell, not very obvious, and a > confusing clash with the also-prominent PyPy project. Why not call > the place for eggs just eggs? I.e., http://eggs.python.org/ +1 -- nice! > THE COMMAND LINE PACKAGE MANAGEMENT TOOL > > The "egg" command should enable you to at least find, show info for, > install, and uninstall packages. I think the most common way to do > command line tools like this is to offer sub-commands, a la, bzr, > port, svn, apt-get, gem, so I suggest: > > egg - list out a help of commands > egg search - search for eggs (aliases: find/list) > egg info - show info for egg (aliases: show/details) > egg install - install named eggs > egg uninstall - uninstall eggs (aliases: remove/purge/delete) > > so you can do: > > egg search bittorrent > > to find all packages that have anything to do with bittorrent (full- > text search of the package index), and then: > > egg install iTorrent > > to actually download and install the package. Yes, yes, yes, +1. > Optional: These days XMLRPC (and the WS-Deathstar) seems to be losing > steam to REST, so I think we'd gain a lot of "hackability" by > enabling a REST interface for accessing packages. +1 > Eventually we probably need to enforce package signing. +1 > It'd be good for "egg" to support both system- and user-wide > configurations, and to support downloading from several package > indexes, like apt-get does. And it would be nice if Python could be adapted to provide for user- specific site-packages so that PYTHONPATH hackery isn't necessary. Bjorn, I wish I had time to help, but I like where you're going with this. I think it would greatly improve the utility of eggs. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRsSpsXEjvBPtnXfVAQIQnwP9FibXQYRMlhG9VScTbkr1lKB84k0+Awl8 NFIvl+h8ADkiItJsAmGYlCRO/dAUgE9imKoPD4Z35LbVvz9y6oiTRU6KYJwFossk ytIYBLQTf+727NQD4860+1Q23O1mFwf612/M4W4niO6H7GDCVZnxbSFJZIoaYNcH VUBp4F8WAy0= =AYko -----END PGP SIGNATURE----- From bjorn at exoweb.net Fri Aug 17 02:18:46 2007 From: bjorn at exoweb.net (=?ISO-8859-1?Q?Bj=F8rn_Stabell?=) Date: Fri, 17 Aug 2007 08:18:46 +0800 Subject: [Distutils] Simpler Python package management: the "egg" command In-Reply-To: <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> Message-ID: <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> On Aug 17, 2007, at 01:12, Lu?s Bruno wrote: > Bj?rn Stabell: [...] >> egg info pysqlite - show information about >> pysqlite >> egg info pysqlite/2.0.0 - show information about version >> 2.0.0 of pysqlite >> egg sync local - rescan local packages and >> update cache db > > Sorry, but I think you meant apt-get instead of egg. No, I didn't > search the archives. But making an apt-get repository (yum, emerge...) > can't be *that* hard; it also can't be an uncommon idea. Someone must > have suggested it before. The "egg" prototype already does the above commands. > On second thought, if I recall correctly Debian-style repositories > have to update a master Packages catalog for *each* and *every* > *single* upload. That's a -1. I think you've asked for a "sync local" > master and I snipped it. Any other -1? > > We'll, getting the repositores updated for each single upload becomes > O(N), but it's a small N anyway. One per supported repository format. The "egg sync" stuff was to get the latest package information from PyPI and from your locally installed packages so that you can do fast and offline queries against it. If you don't sync, you'll have to rescan every time; sync'ing is just an optimization, and since it gets put it in a little database, it makes making queries etc much easier as another benefit. [...] >> Perhaps "egg" should provide a simple web server to allow browsing >> (and perhaps installation from) local packages. > > D*mn. Right now you just serve your .../site-packages and you can > easy_install from it (I think Phillip Eby said as much recently). I haven't seen that done, but since eggs in uninstalled and installed form are the same, it should be easy. Rgds, Bjorn From eu at lbruno.org Fri Aug 17 11:51:45 2007 From: eu at lbruno.org (Luis Bruno) Date: Fri, 17 Aug 2007 10:51:45 +0100 Subject: [Distutils] [Catalog-sig] PyPI - Evolve our own or reuse existing package systems? In-Reply-To: <88d0d31b0708161120u54fc1d38ib50366d2125acc7c@mail.gmail.com> References: <200708150037.57652.paul@boddie.org.uk> <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> <865235D6-2674-4B78-ABE8-4DFC5919576E@exoweb.net> <88d0d31b0708161120u54fc1d38ib50366d2125acc7c@mail.gmail.com> Message-ID: <7555ca2e0708170251s62e5fc84me5de6589a13001c2@mail.gmail.com> Hello there, Jeff Shell wrote: > This goes into a deeper frustration I've had in the past: I installed > MySQL on my desktop (Mac OS X) using a disk image / .pkg installer > downloaded from MySQL's web site. Then I think I tried installing a > python package from MacPorts (maybe just the mysql bindings?) that had > a MySQL dependency. It didn't detect that I already had MySQL > installed, and MacPorts then tried installing it on its own. IIRC there are variants in some MacPorts that removed dependencies from their setup.py-like file and used the system ones. It can also be dealt with by creating phantom-packages which provide the virtual "name" mysql-client, for example (which is how it's been done in apt-get repositories, etc, etc.). > Even on the most barren of machines, it's relatively > easy to build and install Python from source. I can agree with that. From eu at lbruno.org Fri Aug 17 11:59:26 2007 From: eu at lbruno.org (Luis Bruno) Date: Fri, 17 Aug 2007 10:59:26 +0100 Subject: [Distutils] Simpler Python package management: the "egg" command In-Reply-To: <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> Message-ID: <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.com> Hello again, I'm trying to ASCII-fy so that I don't send another base64 blob. Bjorn Stabell wrote: > Luis Bruno wrote: > > Bjorn Stabell wrote: > > > egg info pysqlite > > > egg info pysqlite/2.0.0 > > > egg sync local > > > > Sorry, but I think you meant apt-get instead of egg. No, I didn't > > search the archives. But making an apt-get repository (yum, emerge...) > > can't be *that* hard; it also can't be an uncommon idea. Someone must > > have suggested it before. > > The "egg" prototype already does the above commands. I didn't have a look around your code. My whole post could be summarized as: you're reinventing the apt-get paraphernalia. I'd prefer to drop a python.list into /etc/apt/sources.list.d/ with: deb http://eggs.python.org/apt And use the rest of the tools I already have. The really *big* -1 this has is that I'm basically gonna be using --single-version-externally-managed eggs (which makes it impossible to have multiple "inactive" versions and require() them, if I understood Phillip Eby correctly). > The "egg sync" stuff was to get the latest package information from > PyPI and from your locally installed packages so that you can do fast > and offline queries against it. If you don't sync, you'll have to > rescan every time; sync'ing is just an optimization, and since it > gets put it in a little database, it makes making queries etc much > easier as another benefit. I was thinking "sync local" re-gets the repository's Packages master-list. Then you read in the locally installed ones (which is a matter of traversing sys.path and looking for the .egg-info files; I think those are now (as of 2.5) expected to be there. All this has been done before; that's why I'm being so bone-headed about it. > > D*mn. Right now you just serve your .../site-packages and you can > > easy_install from it. > > I haven't seen that done, but since eggs in uninstalled and installed > form are the same, it should be easy. I think easy_install -f can work against an Apache directory index. I thought that was the whole point behind it, really. -- Luis "Bone-headed describes me so well" Bruno From arve.knudsen at gmail.com Fri Aug 17 12:47:21 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Fri, 17 Aug 2007 12:47:21 +0200 Subject: [Distutils] [Catalog-sig] PyPI - Evolve our own or reuse existing package systems? In-Reply-To: <865235D6-2674-4B78-ABE8-4DFC5919576E@exoweb.net> References: <200708150037.57652.paul@boddie.org.uk> <212C47AD-A14B-4D81-B6DE-3AEF13846D95@exoweb.net> <865235D6-2674-4B78-ABE8-4DFC5919576E@exoweb.net> Message-ID: Very glad to hear you're interested in my system Bj?rn. On 8/16/07, Bj?rn Stabell wrote: > > On Aug 15, 2007, at 21:34, Arve Knudsen wrote: > > These are some interesting points you are making. I have in fact > > been developing a general software deployment system, Conduit, in > > Python for some time, that is capable of supporting several major > > platforms (at the moment: Linux, Windows and OS X). It's not > > reached any widespread use, but we (at the Simula Research > > Laboratory) are using it to distribute software to students > > attending some courses at the university of Oslo. Right now we are > > in the middle of preparing for the semester start, which is next week. > > > > The system is designed to be general, both with regard to the > > target platform and the deployable software. I've solved the > > distribution problem by constructing an XML-RPC Web service, that > > serves information about software projects in RDF (based on the > > DOAP format). This distribution service is general and independent > > of the installation system, which acts as a client of the latter. > > > > If this sounds interesting to you I'd love if you checked it out > > and gave me some feedback. It is an experimental project, and as > > such we are definitely interested in ideas/help from others. > > Hi Arve, > > That's an interesting coincidence! :) > > Without turning it into a big research project, it would be > interesting to hear what you (honestly) thought were the strengths > and weaknesses of Conduit compared to, say, deb/rpm/ports/emerge, > whichever you have experience with. I did download and look at > Conduit, but haven't tried it yet. I would say the main difference lies in how Conduit is designed to be a completely general solution for distributing software and deploying it on user's systems, with as loose coupling as possible. You could say that what I am trying to achieve is closer to MacroVision's Install Anywhere / Flexnet Connect than to monolithic package managers such as APT, Emerge etc. The former offers a complete solution to independent providers for letting them deliver software and maintain it (with updates) over time, while the latter is a tightly integrated service which is even used to implement operating systems (e.g. Debian, Gentoo). Conduit tries to offer the best of both worlds by building a central software portal from independent project representations. The idea is that software providers maintain their own profile within the portal service, and associate with this a number of projects which are described in RDF (an extension of the DOAP vocabulary). The portal service accumulates these data, and expose them to installation agents via a public XML-RPC API. I've written a framework for Conduit agents that currently supports installing on Linux, Windows (XP/Vista) and OS X. I find it a great strength to be able to offer a common installation system for all three platforms, but the weakness is that generally it doesn't integrate as well with the operating systems as native installers do. On Windows at least I plan to piggy-back on the native installation service (Windows Installer), to achieve better integration without having to reinvent the wheel. On Linux it is worse since there is no well-defined native installation service, but instead a bunch of different packaging systems which overlap with my own deployment model (specification of dependencies etc.). There are so many ways to take this, and so many "strategic" > decisions that I'd hope people on the list could help out with. > > Personally I think it would be great if we had a strong Python-based > central package system, perhaps based on Conduit. I'm pretty sure > Conduit would have to have the client and server-side components even > more clearly separated, though, and the interface between them open > and clearly defined (which I think it is, but it would have to be > discussed). The client and server components should be clearly separated as-is, but the server API should definitely be reviewed and properly defined. Conduit-specific support exists on the server as extensions (namespace "conduit") of the RDF vocabulary. I see Conduit (and PyPI) supports DOAP, and looking around I also > found http://python.codezoo.com/ by O'Reilly; it also seems to have a > few good ideas, for example voting and some quality control (although > that's a very difficult decision, I guess). CodeZoo is a very interesting initiative. I let myself inspire in part by CodeZoo when I started designing Conduit, but mostly by SWED ( http://swed.org.uk) which has a similar model of accumulating decentralized information in RDF for centralized access (via a Web interface). I would actually like Conduit's distribution service to evolve into something with similar functionality to CodeZoo. A rich Web interface for navigating the catalog of software would be awesome (an alternative to sourceforge?). I've also pondered the possibility of user profiles in the portal so that one can keep preferences centrally, for instance as a way to define personal "installation sets" (e.g., after installing a new Linux, restore your previous installations). Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070817/69e3cfbd/attachment.html From bwinton at latte.ca Fri Aug 17 15:07:20 2007 From: bwinton at latte.ca (Blake Winton) Date: Fri, 17 Aug 2007 09:07:20 -0400 Subject: [Distutils] Simpler Python package management: the "egg" command In-Reply-To: <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.com> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.com> Message-ID: <46C59D88.1040303@latte.ca> Luis Bruno wrote: >>>> egg info pysqlite >>>> egg info pysqlite/2.0.0 >>>> egg sync local >>> Sorry, but I think you meant apt-get instead of egg. >> The "egg" prototype already does the above commands. > I'd prefer to drop a python.list into /etc/apt/sources.list.d/ with: > deb http://eggs.python.org/apt > And use the rest of the tools I already have. Me too! Uh, except the closest thing I have to /etc/apt/sources.list.d/ is C:\Program Files\Apt\sources.list.d\... And I don't have any tools that will work with it. Fortunately, "egg info pysqlite" and "egg sync local" should work out just fine for me. :) Later, Blake. From me at lbruno.org Mon Aug 20 15:58:05 2007 From: me at lbruno.org (Luis Bruno) Date: Mon, 20 Aug 2007 14:58:05 +0100 Subject: [Distutils] Simpler Python package management: the "egg" command In-Reply-To: <46C59D88.1040303@latte.ca> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.com> <46C59D88.1040303@latte.ca> Message-ID: <7555ca2e0708200658q7584d9c3vd0cf89305c131ab5@mail.gmail.com> Blake Winton wrote: > Me too! Uh, except the closest thing I have to /etc/apt/sources.list.d/ > is C:\Program Files\Apt\sources.list.d\... And I don't have any tools > that will work with it. Good point; I had forgotten there isn't a Windows fetch-X-from-repositories. The closest thing that comes to mind is the whole Policy Object enchilada which I never really understood. -- Luis "and I gotta +1 your marvelous use of irony" Bruno From sklein at cpcug.org Mon Aug 20 18:30:44 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 20 Aug 2007 12:30:44 -0400 (EDT) Subject: [Distutils] I finally got the kiva rpm to build using bdist_rpm In-Reply-To: <46C51194.1050108@enthought.com> References: <15078.207.188.248.157.1187306093.squirrel@www.cpcug.org> <46C4D697.9040208@enthought.com> <52093.71.163.205.195.1187314144.squirrel@www.cpcug.org> <46C51194.1050108@enthought.com> Message-ID: <15542.207.188.248.157.1187627444.squirrel@www.cpcug.org> I finally got the kiva rpm to build. I also got the docs and examples into the proper doc directory. Here is what it took: First, run "python setup.py bdist_rpm", which will fail with a list of installed but unpackaged files. Somehow the full set of installed files isn't being recorded, probably because of some problem among numpy distutils, python distutils, setuptools, and Fedora rpm brp-python-bytecompile that isn't solved by the [install] optimize=1 option in setup.cfg (that works in most cases). Then create a file (I called it problem_files) with a list of the installed but unpackaged files. Also, create a new install script to be put into the rpm spec file that appends the problem file list to INSTALLED_FILES (where the installed files are recorded in the default spec file). Then run the setup with options that use the new install script. The rpm packaging of the docs files is just another option to the setup run. Below are the relevant scripts and files. I think I may still have a problem with the enthought namespace __init__.py and the nspkg.pth file that will presumably be fixed in an upcoming version of setuptools. BTW, I don't know what effect this has, but there are also __init__.pyc and __init__.pyo files packaged into the rpm with the namespace file. IIRC it may not be supposed to be there, but it got in somehow, probably because of the same problem that caused the installed files to be incorrectly recorded. Stan Klein ------------------------------------------------------------------------------------- The setup command line: python setup.py bdist_rpm --doc-files=" examples/ docs/ " --install=install_script2 install_script2 is: python setup.py install --single-version-externally-managed --root= $RPM_BUILD_ROOT --record=INSTALLED_FILES cat /home/stan/new-enthought-svn/enthought.branches/enthought.kiva_2.0/problem_files >> INSTALLED_FILES problem_files is: /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/join_stroke_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/join_stroke_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/points_in_polygon_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/points_in_polygon_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/rgba_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/rgba_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/stroke_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/stroke_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_arc.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_arc.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_draw_dash.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_draw_dash.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_ft.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_ft.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_graphics_context_system.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_graphics_context_system.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_image3.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_image3.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_save.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_save.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_utils.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_utils.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/unicode_font_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/unicode_font_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/affine_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/affine_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/basecore2d_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/basecore2d_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/mac_simpletest.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/mac_simpletest.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/macport_test.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/macport_test.pyo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070820/95efb1b4/attachment.htm From martin at v.loewis.de Mon Aug 20 18:26:19 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 20 Aug 2007 18:26:19 +0200 Subject: [Distutils] [Catalog-sig] Simpler Python package management: the "egg" command In-Reply-To: <7555ca2e0708200658q7584d9c3vd0cf89305c131ab5@mail.gmail.com> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.com> <46C59D88.1040303@latte.ca> <7555ca2e0708200658q7584d9c3vd0cf89305c131ab5@mail.gmail.com> Message-ID: <46C9C0AB.8070204@v.loewis.de> > Good point; I had forgotten there isn't a Windows > fetch-X-from-repositories. The closest thing that comes to mind is the > whole Policy Object enchilada which I never really understood. That works actually very well, but requires that computers are domain members. Then, you can deploy selected software on a selected subset of the machines in the domain, or for a selected subset of the users. Regards, Martin From pje at telecommunity.com Mon Aug 20 19:38:31 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Aug 2007 13:38:31 -0400 Subject: [Distutils] I finally got the kiva rpm to build using bdist_rpm In-Reply-To: <15542.207.188.248.157.1187627444.squirrel@www.cpcug.org> References: <15078.207.188.248.157.1187306093.squirrel@www.cpcug.org> <46C4D697.9040208@enthought.com> <52093.71.163.205.195.1187314144.squirrel@www.cpcug.org> <46C51194.1050108@enthought.com> <15542.207.188.248.157.1187627444.squirrel@www.cpcug.org> Message-ID: <20070820173608.C042A3A408D@sparrow.telecommunity.com> At 12:30 PM 8/20/2007 -0400, Stanley A. Klein wrote: >problem_files is: > >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyc >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyo >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyc >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyo >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyc >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyo >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyc >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyo >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyc >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyo >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyc >/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyo Is kiva.agg.tests a package? If so, is it listed in setup()? If not, are the individual modules listed as modules to be installed? If these are just .py files, that would explain why they're not listed in INSTALLED_FILES. From pje at telecommunity.com Mon Aug 20 19:44:17 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Aug 2007 13:44:17 -0400 Subject: [Distutils] trouble making egg... In-Reply-To: <46C16BD9.2000108@simplistix.co.uk> References: <46BF74BD.2000509@simplistix.co.uk> <20070812214346.69B533A403D@sparrow.telecommunity.com> <46C0286A.3070404@simplistix.co.uk> <20070813161442.8820A3A40A8@sparrow.telecommunity.com> <46C16BD9.2000108@simplistix.co.uk> Message-ID: <20070820174154.EE2B93A408D@sparrow.telecommunity.com> At 09:46 AM 8/14/2007 +0100, Chris Withers wrote: >Phillip J. Eby wrote: >>At 10:46 AM 8/13/2007 +0100, Chris Withers wrote: >>>Phillip J. Eby wrote: >>>>>Given the component.xml in there, I'm pretty sure I need to add a >>>>>zip_safe=True, but what else do I need to do to get setup.py to >>>>>correctly build an egg and a source distro? >>>>Short answer: move the package and any data files that should be >>>>included to a mailinglogger/ subdirectory. >>> >>>What ends up in the root folder then? just the setup.py? >>and anything else you want in the source distro, like the docs and >>examples. I also usually throw my tests there, and don't >>distribute them with the egg, but preferences vary. > >Okay, I think I have this now: > >https://secure.simplistix.co.uk/svn/Simplistix/mailinglogger/tags/3.2.1/ > >However, when I do: > >python2.4 setup.py sdist bdist_egg upload > >...none of the package metadata gets uploaded: > >http://pypi.python.org/pypi?:action=display&name=mailinglogger&version=3.2.1 > >This used to work: > >http://pypi.python.org/pypi?:action=display&name=mailinglogger&version=3.2.0 > >...and I can't see any changes in my setup.py that might be relevent. > >Any ideas? Is there a problem with PyPi at the moment? No idea; but it looks correct to me now. From pje at telecommunity.com Mon Aug 20 19:52:04 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Aug 2007 13:52:04 -0400 Subject: [Distutils] Finding eggs when dependency_links and user specified URLs both given to easy_install? In-Reply-To: <46C499B3.8020500@enthought.com> References: <46C499B3.8020500@enthought.com> Message-ID: <20070820174943.92F383A408D@sparrow.telecommunity.com> At 01:38 PM 8/16/2007 -0500, Dave Peterson wrote: >What happens if a user specifies a URL via a '-f' option on the >easy_install command line but our published eggs specify >dependency_links in the setup.py? Will easy_install search the user's >specified location first, last, or will it not search them at all? >(I'm hoping for the first!) The precedence is determined by the versions and urls specifically being searched, not by the order in which -f options are specified. The user's -f options will be searched first, but the eggs' dependency_links will also be searched. >But early indications are that when a user installs by doing something like: > easy_install -f ets==2.5b2 >then the first egg gets found properly in the platform specific repo, >but all of its dependencies are only being searched for in the source >repo. Is this just a misinterpretation of the output generated by >easy_install? Probably; each new set of dependency links is just treated as if they were added to the -f list. And the binary eggs should be given preference to the source tarballs, assuming that they are for the same target version, or you explicitly request a specific version. >BTW, it would be nice if there was an option to name your generated eggs >based on a full specification of OS and version, hardware arch, etc. >And of course, the whole setuptools universe of tools would need to know >to search for eggs with that level of accuracy before dropping down to >more generic specs. The optimal spelling of platform strings is an open issue. Unfortunately, I don't have any good solutions, and nobody has stepped forward with a solid design proposal. >Another similar use case: Some of our users want to locally cache eggs >for distribution within their corporate environment for various reasons >(eggs tested and approved to work with their corporate desktop standard, >etc.) Those users would also be specifying a '-f' option to >easy_install to point at their corporate cache. However, all of our >eggs will have dependency_links specified. Will these corporate users >be able to install and pull things from their local repository, or will >easy_install always look at our dependency_links' specified locations >first? The precedence system doesn't work this way; distributions have precedence, --find-links and dependency_links don't. So, unless they are requesting a specific version, a newer source version on your servers will take precedence over an older binary version on their local system. From pje at telecommunity.com Mon Aug 20 20:01:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Aug 2007 14:01:02 -0400 Subject: [Distutils] [Catalog-sig] Simpler Python package management: the "egg" command In-Reply-To: <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.co m> References: <8CD37324-80E0-4C56-8F76-E313328B1CFD@exoweb.net> <5500FDAF-34F5-4075-9ED7-2C849CA1F266@exoweb.net> <7555ca2e0708161012j25b957d4q1f1a04c73715b7ec@mail.gmail.com> <7C93A25C-B7B9-471C-AFF3-437D551DF282@exoweb.net> <7555ca2e0708170259v1f946099nea35e00412d7fdbb@mail.gmail.com> Message-ID: <20070820175840.63A683A408D@sparrow.telecommunity.com> At 10:59 AM 8/17/2007 +0100, Luis Bruno wrote: >The really *big* -1 this has is that I'm basically gonna be using >--single-version-externally-managed eggs (which makes it impossible to >have multiple "inactive" versions and require() them, if I understood >Phillip Eby correctly). You can have inactive versions and require() them, they just have to be .egg files or directories. You can have a "default" version that's installed --single-version, e.g. by a system package manager such as RPM. >I was thinking "sync local" re-gets the repository's Packages >master-list. Then you read in the locally installed ones (which is a >matter of traversing sys.path and looking for the .egg-info files; I >think those are now (as of 2.5) expected to be there. Please, please, *please* use the published APIs in pkg_resources for this. Too many people are writing tools that inspect egg files and directories directly -- and get it only partly right, making assumptions about the formats that aren't valid across platforms, Python versions, etc., etc. In general, if you are doing absolutely *anything* with on-disk formats of eggs, and you didn't read enough of the docs to find the equivalent APIs, it's a near-certainty that you don't understand the format well enough to write your own versions. Meanwhile, pkg_resources is proposed for inclusion in the Python 2.6 stdlib, so it's not like it's going to be hard to get a hold of. In this particular example, by the way, if you want to find all locally installed packages, you probably want to be using an Environment instance, which indexes all installed packages by package name, and gives you objects you can inspect in a variety of ways, including using .get_metadata('PKG-INFO') calls to read the .egg-info files -- or .egg-info/PKG-INFO, or EGG-INFO/PKG-INFO, or whatever file is actually involved. (This is why you need to use the API -- there are a lot of devils in the details.) >I think easy_install -f can work against an Apache directory >index. Yes. >I thought that was the whole point behind it, really. One of them, anyway. There are other aspects besides -f that work for directory indexes, such as PyPI "home page" and "download" URL links. From jim at zope.com Mon Aug 20 19:18:49 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 20 Aug 2007 13:18:49 -0400 Subject: [Distutils] Heads up: major change in zc.buildout policy for selecting distributions Message-ID: I'm about to make a new release of zc.buildout that uses a different policy for selecting distributions. In particular, by default, zc.buildout will now prefer final distributions over non-final ones. If there are final and non-final distributions that satisfy a requirement, buildout will, by default, select a final distribution even if there are non-final distributions that satisfy a requirement. This release will cause many existing buildouts to use older distributions than they do now. You can change this behavior by providing a prefer-final buildout option: [buildout] prefer-final = false Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From chris at simplistix.co.uk Mon Aug 20 21:12:13 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 20 Aug 2007 20:12:13 +0100 Subject: [Distutils] trouble making egg... In-Reply-To: <20070820174154.EE2B93A408D@sparrow.telecommunity.com> References: <46BF74BD.2000509@simplistix.co.uk> <20070812214346.69B533A403D@sparrow.telecommunity.com> <46C0286A.3070404@simplistix.co.uk> <20070813161442.8820A3A40A8@sparrow.telecommunity.com> <46C16BD9.2000108@simplistix.co.uk> <20070820174154.EE2B93A408D@sparrow.telecommunity.com> Message-ID: <46C9E78D.4010502@simplistix.co.uk> Phillip J. Eby wrote: >> Any ideas? Is there a problem with PyPi at the moment? > > No idea; but it looks correct to me now. I had to do: python2.4 setup.py sdist bdist_egg register upload ...which seems a bit weird. So I have to do 'register' every time in order to get the metadata up there? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From miles at jamkit.com Mon Aug 20 22:42:00 2007 From: miles at jamkit.com (Miles) Date: Mon, 20 Aug 2007 21:42:00 +0100 Subject: [Distutils] zc.buildout In-Reply-To: References: Message-ID: <46C9FC98.7070001@jamkit.com> Thanks for the info. I've got the extends stuff working nicely for having a "library" of tasks, and then project-specific bits in a buildout, but haven't got it structured for specific different setups yet (testing, deployment, devel, etc). I just discovered how to turn "off" the recipe used for a part when overriding and just provide configuration (e.g. for dev, don't build python, just provide a location for the one already installed) so that removes quite a bit of repetition. Thinking about it a bit more, what originally sparked off the post was a need to run some steps as root (e.g. to install startup scripts in /etc/init.d), and others as a different user (zope files owned by the zope user). At the moment, we tackle this by (for example): $ su $ bin/buildout -c buildout-root.cfg $ exit $ su zope $ bin/buildout -c buidlout-zope.cfg ... However, this starts to get complicated as more users are involved! Plus, each step cannot easily use the configuration from a previous one, so there's some repetition. Do you have a pattern that covers this at all? Miles Martin Aspeli wrote: > Miles wrote: > >>Hi, >> >>We have some a zc.buildout-based deployment system, and we're looking at >>how we can make the process slightly easier for moving between >>development, testing and production environments. >> >>I was wondering if others have a pattern for setting up a buildout in >>this way? I am imagining that this is better done by running the >>different steps, but putting the outputs in different locations (e.g. >>for testing, put apache configurations in some directory in the test >>tree), rather than trying to selectively run steps (e.g. for testing, i >>do not want apache configurations set up). Is there a useful >>buildout.cfg pattern that covers this? >> >>Also, one other use-case that we've got is to be able to group parts >>together into sets. So, for example, I'd lie to be able to do something >>like: >> >>[buildout] >>parts = awstats >> >>[awstats] >>parts = awstats-download apacheconf-install crontab-install >> >>which would tell buildout to run the three parts listed, rather than >>remembering all 3 specific parts in the parts list. The advantage being >>to ensure the bits are always run together. >> >>Has anyone done anything like this at all? > > > I'm not sure it meets your needs exactly, but you can have one buildout > file depend on another. In this case, they are "merged": > > [buildout] > base = basics.cfg > > ... > > Here, you get all of basics.cfg, and subsequent parts of the file can > override or extend those settings. Sections with the same name are > merged as well, so you can override and merge parts of a part, as it were. > > So, for example, you may have a buildout.cfg that has your normal > development setup, and a deployment.cfg that is based on this, but turns > off debugging and adds deployment stuff like Apache. Use -c when running > ./bin/buildout to specify an alternate file. > > Martin > From miles at jamkit.com Mon Aug 20 22:42:00 2007 From: miles at jamkit.com (Miles) Date: Mon, 20 Aug 2007 21:42:00 +0100 Subject: [Distutils] zc.buildout In-Reply-To: References: Message-ID: <46C9FC98.7070001@jamkit.com> Thanks for the info. I've got the extends stuff working nicely for having a "library" of tasks, and then project-specific bits in a buildout, but haven't got it structured for specific different setups yet (testing, deployment, devel, etc). I just discovered how to turn "off" the recipe used for a part when overriding and just provide configuration (e.g. for dev, don't build python, just provide a location for the one already installed) so that removes quite a bit of repetition. Thinking about it a bit more, what originally sparked off the post was a need to run some steps as root (e.g. to install startup scripts in /etc/init.d), and others as a different user (zope files owned by the zope user). At the moment, we tackle this by (for example): $ su $ bin/buildout -c buildout-root.cfg $ exit $ su zope $ bin/buildout -c buidlout-zope.cfg ... However, this starts to get complicated as more users are involved! Plus, each step cannot easily use the configuration from a previous one, so there's some repetition. Do you have a pattern that covers this at all? Miles Martin Aspeli wrote: > Miles wrote: > >>Hi, >> >>We have some a zc.buildout-based deployment system, and we're looking at >>how we can make the process slightly easier for moving between >>development, testing and production environments. >> >>I was wondering if others have a pattern for setting up a buildout in >>this way? I am imagining that this is better done by running the >>different steps, but putting the outputs in different locations (e.g. >>for testing, put apache configurations in some directory in the test >>tree), rather than trying to selectively run steps (e.g. for testing, i >>do not want apache configurations set up). Is there a useful >>buildout.cfg pattern that covers this? >> >>Also, one other use-case that we've got is to be able to group parts >>together into sets. So, for example, I'd lie to be able to do something >>like: >> >>[buildout] >>parts = awstats >> >>[awstats] >>parts = awstats-download apacheconf-install crontab-install >> >>which would tell buildout to run the three parts listed, rather than >>remembering all 3 specific parts in the parts list. The advantage being >>to ensure the bits are always run together. >> >>Has anyone done anything like this at all? > > > I'm not sure it meets your needs exactly, but you can have one buildout > file depend on another. In this case, they are "merged": > > [buildout] > base = basics.cfg > > ... > > Here, you get all of basics.cfg, and subsequent parts of the file can > override or extend those settings. Sections with the same name are > merged as well, so you can override and merge parts of a part, as it were. > > So, for example, you may have a buildout.cfg that has your normal > development setup, and a deployment.cfg that is based on this, but turns > off debugging and adds deployment stuff like Apache. Use -c when running > ./bin/buildout to specify an alternate file. > > Martin > From sklein at cpcug.org Mon Aug 20 23:15:38 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Mon, 20 Aug 2007 17:15:38 -0400 (EDT) Subject: [Distutils] I finally got the kiva rpm to build using bdist_rpm In-Reply-To: <20070820173608.C042A3A408D@sparrow.telecommunity.com> References: <15078.207.188.248.157.1187306093.squirrel@www.cpcug.org> <46C4D697.9040208@enthought.com> <52093.71.163.205.195.1187314144.squirrel@www.cpcug.org> <46C51194.1050108@enthought.com> <15542.207.188.248.157.1187627444.squirrel@www.cpcug.org> <20070820173608.C042A3A408D@sparrow.telecommunity.com> Message-ID: <15693.207.188.248.157.1187644538.squirrel@www.cpcug.org> The overall kiva package is built by a combination of numpy distutils and setuptools. There are multiple setup.py files involved because of the needs of numpy distutils for compiling the extensions. Somehow the setup() of the main setup.py is used by both numpy distutils and setuptools. The relevant import statements are import setuptools from numpy.distutils.core import setup The tests directory under kiva is referenced in the setup.py configuration() there as config.add_data_dir('tests'). There is a similar statement in the setup.py file under kiva/agg. I tried to move the "tests" directories under both kiva and kiva/agg to the top-level examples directory without doing an svn check-in. That didn't seem to work, I suppose because of setuptools svn awareness. I tried numerous other fixes, some that I came up with on my own and others suggested by Dave Peterson and this is the only one that ran to completion. However, I don't remember everything I tried. One thing I did try was an in-place build (using kivabuild.py that mainly uses numpy distutils) to get the extensions built and a relatively vanilla setup.py file with setuptools to do the install and packaging steps. That approach left the .so files missing from the package. If there is a cleaner way to do the rpm package for kiva, I would prefer to use it, but I haven't found one yet. I got the idea for this workaround approach from a posting somewhere by someone who had a similar problem with rpm packaging. Stan Klein On Mon, August 20, 2007 1:38 pm, Phillip J. Eby wrote: > At 12:30 PM 8/20/2007 -0400, Stanley A. Klein wrote: >>problem_files is: >> >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyc >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyo >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyc >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyo >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyc >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyo >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyc >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyo >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyc >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyo >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyc >>/usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyo > > Is kiva.agg.tests a package? If so, is it listed in setup()? If > not, are the individual modules listed as modules to be installed? > > If these are just .py files, that would explain why they're not > listed in INSTALLED_FILES. > From jim at zope.com Mon Aug 20 23:19:09 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 20 Aug 2007 17:19:09 -0400 Subject: [Distutils] zc.buildout In-Reply-To: References: Message-ID: On Aug 8, 2007, at 1:41 PM, Miles wrote: Sorry to take so long to reply.... Your earlier post today reminded me that I needed to reply. I hope that this will answer both your original post and today's. > Hi, > > We have some a zc.buildout-based deployment system, and we're > looking at > how we can make the process slightly easier for moving between > development, testing and production environments. > > I was wondering if others have a pattern for setting up a buildout in > this way? I am imagining that this is better done by running the > different steps, but putting the outputs in different locations (e.g. > for testing, put apache configurations in some directory in the test > tree), rather than trying to selectively run steps (e.g. for > testing, i > do not want apache configurations set up). Is there a useful > buildout.cfg pattern that covers this? > > Also, one other use-case that we've got is to be able to group parts > together into sets. So, for example, I'd lie to be able to do > something > like: > > [buildout] > parts = awstats > > [awstats] > parts = awstats-download apacheconf-install crontab-install > > which would tell buildout to run the three parts listed, rather than > remembering all 3 specific parts in the parts list. The advantage > being > to ensure the bits are always run together. > > Has anyone done anything like this at all? Yes, In practice, we find we need 2 modes, development, and deployment. (Staging is just a special case of deployment for us and I expect, many others.) We use buildout.cfg for development, which includes most testing. We deploy using RPMs. We have an rpm.cfg and, because we create relocatable RPMs, an rpmpost.cfg that fixes up paths. We also have a separate configuration file that configures software instances, rather than building software. The buildout.cfg, and rpm.cfg, are run by the developer. The later is run via an RPM spec file. The rpmpost.cfg is run automatically by root at part of the software install. This uses buildout on offline mode so that no new software can be installed via the network. Basically, it's in the RPM (or already on the system) or it's not available. This is for security purposes, since the buildout is run as root. After the software is installed, we run the buildout again using yet another configuration file. This configuration file sets up "instances" of the software such as database and application processes. This is run as root and in offline mode. It writes to places like /etc, /var/log, and so on. As Martin suggested, you can use the fact that buildout configurations can extend other buildout configurations to factor commonalities. The various configuration files often extend a common one, but they don't need to. I've been meaning to write this up in detail with an example, but haven't been able to find time to. :( I'll at least put an example together without words so that people can maybe figure out what we've done until I can find the time to write words to go with it. On Aug 20, 2007, at 4:42 PM, Miles wrote: ... > Thinking about it a bit more, what originally sparked off the post > was a > need to run some steps as root (e.g. to install startup scripts in > /etc/init.d), and others as a different user (zope files owned by the > zope user). > > At the moment, we tackle this by (for example): > > $ su > $ bin/buildout -c buildout-root.cfg > $ exit > $ su zope > $ bin/buildout -c buidlout-zope.cfg > ... > > However, this starts to get complicated as more users are involved! Why do you need more users to be involved. In fact, I don't see a need to have more than 2 users involved, the developer and root. > Plus, each step cannot easily use the configuration from a previous > one, > so there's some repetition. This is mostly a matter of factoring. In fact, it often is possible to reuse configuration from previous steps. (There are of course, cases where this can be more convenient.) > Do you have a pattern that covers this at all? See if my description of what we do above makes any sense. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Aug 20 23:55:22 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 20 Aug 2007 17:55:22 -0400 Subject: [Distutils] Heads up: major change in zc.buildout policy for selecting distributions In-Reply-To: References: Message-ID: <4C094940-B651-49D7-916D-5A2E59602D11@zope.com> On Aug 20, 2007, at 1:18 PM, Jim Fulton wrote: > I'm about to make a new release of zc.buildout that uses a > different policy for selecting distributions. In particular, by > default, zc.buildout will now prefer final distributions over non- > final ones. If there are final and non-final distributions that > satisfy a requirement, buildout will, by default, select a final > distribution even if there are non-final distributions that satisfy > a requirement. > > This release will cause many existing buildouts to use older > distributions than they do now. > > You can change this behavior by providing a prefer-final buildout > option: > > [buildout] > prefer-final = false I decided, with some prompting from Martin Aspeli, that this would be too disruptive. The latest release of zc.buildout still supports newest releases. To prefer final releases, you have to set the prefer-final option to true. In zc.buildout version 2, final releases will be prefered by default. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Aug 20 23:57:45 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 20 Aug 2007 17:57:45 -0400 Subject: [Distutils] Heads up: major change in zc.buildout policy for selecting distributions In-Reply-To: References: Message-ID: <6A67BD76-2224-445C-A916-8F8C67ACEC25@zope.com> On Aug 20, 2007, at 1:18 PM, Jim Fulton wrote: > I'm about to make a new release of zc.buildout that uses a > different policy for selecting distributions. In particular, by > default, zc.buildout will now prefer final distributions over non- > final ones. If there are final and non-final distributions that > satisfy a requirement, buildout will, by default, select a final > distribution even if there are non-final distributions that satisfy > a requirement. > > This release will cause many existing buildouts to use older > distributions than they do now. > > You can change this behavior by providing a prefer-final buildout > option: > > [buildout] > prefer-final = false I decided, with some prompting from Martin Aspeli, that this would be too disruptive. The latest release of zc.buildout still supports newest releases. To prefer final releases, you have to set the prefer-final option to true. In zc.buildout version 2, final releases will be prefered by default. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Aug 21 00:47:18 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Aug 2007 18:47:18 -0400 Subject: [Distutils] I finally got the kiva rpm to build using bdist_rpm In-Reply-To: <15693.207.188.248.157.1187644538.squirrel@www.cpcug.org> References: <15078.207.188.248.157.1187306093.squirrel@www.cpcug.org> <46C4D697.9040208@enthought.com> <52093.71.163.205.195.1187314144.squirrel@www.cpcug.org> <46C51194.1050108@enthought.com> <15542.207.188.248.157.1187627444.squirrel@www.cpcug.org> <20070820173608.C042A3A408D@sparrow.telecommunity.com> <15693.207.188.248.157.1187644538.squirrel@www.cpcug.org> Message-ID: <20070820224455.9B2183A4072@sparrow.telecommunity.com> At 05:15 PM 8/20/2007 -0400, Stanley A. Klein wrote: >The overall kiva package is built by a combination of numpy distutils and >setuptools. There are multiple setup.py files involved because of the >needs of numpy distutils for compiling the extensions. Somehow the >setup() of the main setup.py is used by both numpy distutils and >setuptools. The relevant import statements are > >import setuptools >from numpy.distutils.core import setup > >The tests directory under kiva is referenced in the setup.py >configuration() there as config.add_data_dir('tests'). There is a similar >statement in the setup.py file under kiva/agg. Okay, that's why they're not working -- and it doesn't matter whether it's plain distutils, numpy distutils, or setuptools; their compiled forms won't be included in the installation manifest if you list them as data files, because none of those packages will *compile* the .py files -- only the Fedora bytecode compile thing will. The only workarounds I see are to: 1. Install them as legitimate modules, not data files, (in which case distutils/setuptools will compile them and list them in the manifest), OR: 2. Don't give them .py extensions -- they'll get listed, but not compiled by anything. These seem to be the only choices to work around the hostile bytecode compiler script in Fedora. From jim at zope.com Tue Aug 21 15:54:15 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 21 Aug 2007 09:54:15 -0400 Subject: [Distutils] Buildout now uses the simple index. Message-ID: <4B56CB31-0C1E-4FDA-9099-7A680DBE3655@zope.com> As of yesterday's zc.buildout release, buildout now uses the simple package index: http://cheeseshop.python.org/simple by default. (You can cause it to use another index by setting the buildout index option.) This change should provide significantly better performance and will make it easier to use historical releases, because the simple interfaces shows releases that have been hidden from the human interface. It would be nice if easy_install also used the simple interface. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From sklein at cpcug.org Tue Aug 21 16:31:52 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Tue, 21 Aug 2007 10:31:52 -0400 (EDT) Subject: [Distutils] I finally got the kiva rpm to build using bdist_rpm In-Reply-To: <20070820224455.9B2183A4072@sparrow.telecommunity.com> References: <15078.207.188.248.157.1187306093.squirrel@www.cpcug.org> <46C4D697.9040208@enthought.com> <52093.71.163.205.195.1187314144.squirrel@www.cpcug.org> <46C51194.1050108@enthought.com> <15542.207.188.248.157.1187627444.squirrel@www.cpcug.org> <20070820173608.C042A3A408D@sparrow.telecommunity.com> <15693.207.188.248.157.1187644538.squirrel@www.cpcug.org> <20070820224455.9B2183A4072@sparrow.telecommunity.com> Message-ID: <16184.207.188.248.157.1187706712.squirrel@www.cpcug.org> On Mon, 2007-08-20 at 18:47 -0400, Phillip J. Eby wrote: > At 05:15 PM 8/20/2007 -0400, Stanley A. Klein wrote: > >The overall kiva package is built by a combination of numpy distutils and > >setuptools. There are multiple setup.py files involved because of the needs of numpy distutils for compiling the extensions. Somehow the setup() of the main setup.py is used by both numpy distutils and setuptools. The relevant import statements are > > > >import setuptools > >from numpy.distutils.core import setup > > > >The tests directory under kiva is referenced in the setup.py > >configuration() there as config.add_data_dir('tests'). There is a similar > >statement in the setup.py file under kiva/agg. > > Okay, that's why they're not working -- and it doesn't matter whether it's plain distutils, numpy distutils, or setuptools; their compiled forms won't be included in the installation manifest if you list them as data files, because none of those packages will *compile* the .py files -- only the Fedora bytecode compile thing will. > > The only workarounds I see are to: > > 1. Install them as legitimate modules, not data files, (in which case distutils/setuptools will compile them and list them in the manifest), > > OR: > > 2. Don't give them .py extensions -- they'll get listed, but not compiled by anything. > > These seem to be the only choices to work around the hostile bytecode compiler script in Fedora. I tried your first "workaround" and it worked. :-) Dave Peterson had suggested commenting out config.add_data_dir('tests') in the two setup.py files or replacing it with config.add_subpackage('tests'). I thought I had tried both. Obviously, I had tried the first but not the second. :-( Thanks. Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070821/a94fbe22/attachment.htm From thomas at thomas-lotze.de Wed Aug 22 13:28:12 2007 From: thomas at thomas-lotze.de (Thomas Lotze) Date: Wed, 22 Aug 2007 13:28:12 +0200 Subject: [Distutils] Declaring conflicts Message-ID: What's the recommended practice, if any, for declaring conflicts between egg distributions? The use case I have in mind is changing dependencies between packages A and B: * A-1 depends on B-1, but not the other way around, * I want to make B-2 depend on A-2, but not the other way around, * and I want to make sure that if someone has A-1 and B-1 installed, they cannot upgrade only A, because A-2 and B-1 conflict in terms of functionality. AFAICS, nothing prevents people from upgrading only A if only dependencies, but no conflicts are declared. I didn't find any documented way of declaring conflicts, though. -- Thomas From jim at zope.com Wed Aug 22 20:52:39 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 22 Aug 2007 14:52:39 -0400 Subject: [Distutils] easy_install doesn't handle extras properly Message-ID: <6D59CE21-B07E-49D1-8826-AC013B6B4F4E@zope.com> Suppose I have a package x that defines an extra y in version 2. In version 3, I remove the extra. Now, if I use easy_install to request x with extra y. easy_install will get the newest x, version 3, and complain that it has no extra y: (env25)jim at ds9:~/tmp$ easy_install -f /home/jim/tmp/x/dist 'x[y]' Searching for x[y] Best match: x 3 Processing x-3-py2.5.egg creating /home/jim/tmp/env25/lib/python2.5/x-3-py2.5.egg Extracting x-3-py2.5.egg to /home/jim/tmp/env25/lib/python2.5 Adding x 3 to easy-install.pth file Installed /home/jim/tmp/env25/lib/python2.5/x-3-py2.5.egg Processing dependencies for x[y] Traceback (most recent call last): ... pkg_resources.UnknownExtra: x 3 has no such extra feature 'y' If I tell easy_install to install a version < 3, then it gets version 2, which has the extra: (env25)jim at ds9:~/tmp$ easy_install -f /home/jim/tmp/x/dist 'x[y] <3' Searching for x[y]<3 Best match: x 2 Processing x-2-py2.5.egg ... Finished installing x[y]<3 easy_install is supposed to get the best distribution that satisfies a requirement, but it doesn't handle the extra part of the requirement. This is just another example of the burden of supporting the feature. Can we *please* get rid of the extras feature. It has some benefit, but I really don't think the benefit justifies the extra complexity in the model or the burden on tool developers. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From james.mills at au.pwc.com Fri Aug 24 02:49:04 2007 From: james.mills at au.pwc.com (james.mills at au.pwc.com) Date: Fri, 24 Aug 2007 10:49:04 +1000 Subject: [Distutils] Cannot install pygtk with easy_install Message-ID: Hi all, I'm using setuptools-0.6c6 and I run into a problem trying to install pygtk. I am using Python-2.5 on Windows XP D:\>easy_install pygtk Searching for pygtk Reading http://cheeseshop.python.org/pypi/pygtk/ Couldn't find index page for 'pygtk' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://cheeseshop.python.org/pypi/ Reading http://cheeseshop.python.org/pypi/pyGTK/2.0.0 Reading http://www.daa.com.au/~james/software/pygtk/ Reading http://cheeseshop.python.org/pypi/PyGTK/2.10.4 Reading http://www.pygtk.org/ Best match: pygtk 2.10.4 Downloading ftp://ftp.gnome.org/pub/GNOME/sources/pygtk/2.10/pygtk-2.10.4.tar.gz Traceback (most recent call last): File "C:\Python25\Scripts\easy_install-script.py", line 8, in load_entry_point('setuptools==0.6c6', 'console_scripts', 'easy_install')() File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\comm and\easy_install.py", line 1670, in main File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\comm and\easy_install.py", line 1659, in with_ei_usage File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\comm and\easy_install.py", line 1674, in File "C:\Python25\lib\distutils\core.py", line 151, in setup dist.run_commands() File "C:\Python25\lib\distutils\dist.py", line 974, in run_commands self.run_command(cmd) File "C:\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\comm and\easy_install.py", line 211, in run File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\comm and\easy_install.py", line 433, in easy_install File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\pack age_index.py", line 463, in fetch_distribution File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\pack age_index.py", line 449, in find File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\pack age_index.py", line 392, in download File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\pack age_index.py", line 612, in _download_url File "C:\Python25\lib\site-packages\setuptools-0.6 c6-py2.5.egg\setuptools\pack age_index.py", line 622, in _attempt_download File "C:\Python25\lib\rfc822.py", line 384, in __getitem__ return self.dict[name.lower()] KeyError: 'content-type' D:\> cheers james James Mills PricewaterhouseCoopers Australia Office: +61 (7) 3257 8847 Mobile: 0430 165 904 Fax: +61 (7) 3023 0938 james.mills at au.pwc.com http://www.pwc.com/au Please consider the environment before printing this email _________________________________________________________________ Please consider the environment before printing this email. PricewaterhouseCoopers is delighted to be recognised by our clients as the Professional Services Market Leader and Best Accounting Firm (Revenue over $250m) in the BRW-St.George Client Choice Awards 2007. http://www.pwc.com/au/awards PwC's latest news, insights and offerings http://www.pwc.com/au This communication is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking any action in reliance on, this communication by persons or entities other than the intended recipient is prohibited. If you received this in error, please inform PricewaterhouseCoopers ("PwC") immediately by return email and delete the material including all copies from any computer. Liability limited by a scheme approved under Professional Standards Legislation. PwC makes no express or implied representation or warranty that this electronic communication or any attachment is free from computer viruses or other defects or conditions which could damage or interfere with the recipients data, hardware or software. This communication and any attachment may have been modified or otherwise interfered with in the course of transmission. From pje at telecommunity.com Fri Aug 24 05:41:19 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Aug 2007 23:41:19 -0400 Subject: [Distutils] Cannot install pygtk with easy_install In-Reply-To: References: Message-ID: <20070824033856.62CAA3A403D@sparrow.telecommunity.com> At 10:49 AM 8/24/2007 +1000, james.mills at au.pwc.com wrote: >Best match: pygtk 2.10.4 >Downloading >ftp://ftp.gnome.org/pub/GNOME/sources/pygtk/2.10/pygtk-2.10.4.tar.gz > >... > return self.dict[name.lower()] >KeyError: 'content-type' I could've sworn this problem was already fixed, but as it turns out, it wasn't fixed for actual file downloads. I've fixed it in SVN; you can "easy_install setuptools==dev06" to get the fixed version. From james.mills at au.pwc.com Fri Aug 24 06:51:11 2007 From: james.mills at au.pwc.com (james.mills at au.pwc.com) Date: Fri, 24 Aug 2007 14:51:11 +1000 Subject: [Distutils] [setuptools]: Installing deps Message-ID: Hi all, I've followed the documentation for setuptools with regards to installing dependencies and installing setuptools (if it's not instlaled, or upgrading it). I can't however get any of these features of setuptools to work correctly. 1. setuptools is not being installed when users install the application's installer 2. required dependencies arenot being installed either. This is being done on Windows XP machine with Python-2.5.1 I do have ez_setup.py in the same directory as setup.py of the project in question and have the following 2 lines at the top of setup.py: import ez_setup ez_setup.use_setuptools() In my setup.py I have: install_requires = [ "pymills" ], However this is not being installed when the app's installer is run. Also setuptools is not being installed either. Am I missing something ? From what I've read this should _all_ work :) cheers James James Mills PricewaterhouseCoopers Australia Office: +61 (7) 3257 8847 Mobile: 0430 165 904 Fax: +61 (7) 3023 0938 james.mills at au.pwc.com http://www.pwc.com/au Please consider the environment before printing this email _________________________________________________________________ Please consider the environment before printing this email. PricewaterhouseCoopers is delighted to be recognised by our clients as the Professional Services Market Leader and Best Accounting Firm (Revenue over $250m) in the BRW-St.George Client Choice Awards 2007. http://www.pwc.com/au/awards PwC's latest news, insights and offerings http://www.pwc.com/au This communication is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking any action in reliance on, this communication by persons or entities other than the intended recipient is prohibited. If you received this in error, please inform PricewaterhouseCoopers ("PwC") immediately by return email and delete the material including all copies from any computer. Liability limited by a scheme approved under Professional Standards Legislation. PwC makes no express or implied representation or warranty that this electronic communication or any attachment is free from computer viruses or other defects or conditions which could damage or interfere with the recipients data, hardware or software. This communication and any attachment may have been modified or otherwise interfered with in the course of transmission. From pje at telecommunity.com Fri Aug 24 17:52:59 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 24 Aug 2007 11:52:59 -0400 Subject: [Distutils] Cannot install pygtk with easy_install In-Reply-To: References: <20070824033856.62CAA3A403D@sparrow.telecommunity.com> Message-ID: <20070824155033.B21B73A403D@sparrow.telecommunity.com> At 02:34 PM 8/24/2007 +1000, james.mills at au.pwc.com wrote: >When I do a: >python setup.py bdist_wininst > >the resulting installer installs the python package but _does not_ install >setuptools on a system that doesn't have it. > >I've tried to work out what I'm donig wrong for the past couple of hours >now with no luck :/ Would appreciate some help! bdist_wininst (or any bdist_* format other than eggs) doesn't install dependencies. Bundling ez_setup only affects source installs. From optilude at gmx.net Mon Aug 27 01:15:59 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 27 Aug 2007 00:15:59 +0100 Subject: [Distutils] Uninstall Message-ID: Hi all, Someone asked me today, how to uninstall a package that was installed with easy_install. The best I could come up with was, remove the .egg file/directory from site-packages and remove the corresponding line from easy_install.pth. That's pretty lame and error prone. Is there a better way? Martin -- Acquisition is a jealous mistress From pje at telecommunity.com Mon Aug 27 01:46:03 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 26 Aug 2007 19:46:03 -0400 Subject: [Distutils] Uninstall In-Reply-To: References: Message-ID: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> At 12:15 AM 8/27/2007 +0100, Martin Aspeli wrote: >Hi all, > >Someone asked me today, how to uninstall a package that was installed >with easy_install. The best I could come up with was, remove the .egg >file/directory from site-packages and remove the corresponding line from >easy_install.pth. That's pretty lame and error prone. > >Is there a better way? Somewhat better: http://peak.telecommunity.com/DevCenter/EasyInstall#uninstalling-packages From optilude at gmx.net Mon Aug 27 01:51:43 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 27 Aug 2007 00:51:43 +0100 Subject: [Distutils] Uninstall In-Reply-To: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> References: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 12:15 AM 8/27/2007 +0100, Martin Aspeli wrote: >> Hi all, >> >> Someone asked me today, how to uninstall a package that was installed >> with easy_install. The best I could come up with was, remove the .egg >> file/directory from site-packages and remove the corresponding line from >> easy_install.pth. That's pretty lame and error prone. >> >> Is there a better way? > > Somewhat better: > > http://peak.telecommunity.com/DevCenter/EasyInstall#uninstalling-packages So ... that removes it from easy_install.pth, but doesn't remove the downloaded .egg file/directory? Martin -- Acquisition is a jealous mistress From optilude at gmx.net Mon Aug 27 01:59:39 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 27 Aug 2007 00:59:39 +0100 Subject: [Distutils] Uninstall In-Reply-To: References: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> Message-ID: Martin Aspeli wrote: > Phillip J. Eby wrote: >> At 12:15 AM 8/27/2007 +0100, Martin Aspeli wrote: >>> Hi all, >>> >>> Someone asked me today, how to uninstall a package that was installed >>> with easy_install. The best I could come up with was, remove the .egg >>> file/directory from site-packages and remove the corresponding line from >>> easy_install.pth. That's pretty lame and error prone. >>> >>> Is there a better way? >> Somewhat better: >> >> http://peak.telecommunity.com/DevCenter/EasyInstall#uninstalling-packages > > So ... that removes it from easy_install.pth, but doesn't remove the > downloaded .egg file/directory? ... mmm, I guess so, in a roundabout kind of way. I think it'd be a huge improvement if easy_install had a '--remove' option or something like that. Even if it just did this exact operation under the hood, it'd be a lot more intuitive. It'd be better if there was a way to remove the scripts and other egg files as well. I don't think people can be expected to realise where eggs are getting installed (i.e. site-packages/foo.egg, and maybe scripts/foobar and whatever else) and find and remove these files. Would easy_install in theory have enough context to work out what files was installed for an egg in the first place? Martin -- Acquisition is a jealous mistress From lxander.m at gmail.com Mon Aug 27 12:11:16 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Mon, 27 Aug 2007 06:11:16 -0400 Subject: [Distutils] Uninstall In-Reply-To: References: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> Message-ID: <525f23e80708270311v4c3b93c2rc9e4e6fe59622984@mail.gmail.com> On 8/26/07, Martin Aspeli wrote: > Martin Aspeli wrote: > I think it'd be a huge improvement if easy_install had a '--remove' > option or something like that. Even if it just did this exact operation > under the hood, it'd be a lot more intuitive. > > It'd be better if there was a way to remove the scripts and other egg > files as well. I don't think people can be expected to realise where > eggs are getting installed (i.e. site-packages/foo.egg, and maybe > scripts/foobar and whatever else) and find and remove these files. > > Would easy_install in theory have enough context to work out what files > was installed for an egg in the first place? As far as I was able to determine, the logic can be recreated, but I couldn't find a record of the user chosen script directory (chosen w/easy_install --script-dir). You could search on PATH, assuming the user has put the script-dir on their PATH, but this starts to get fragile (you might want to look in the file to be sure it is something easy_install created, etc.). You might also run into some issues with projects that have been given egg-info, but not installed by easy_install, not sure on this one. The route I chose to manage uninstalls is to use the --record option to drive an uninstall script. Also fragile because I need to run easy_install my way to make sure the record is generated, but otherwise feels safer to me. I heartily second the need to do this at a more fundamental level. Even if it is just unsinstalling without regard to dependencies. Ultimately, it would be nice to "rotate" the installs, keep just the latest, remove a project and all its dependencies that aren't dependent on anohter installed project, etc., etc. But the first step is being able to install a single project w/o dependencies. From optilude at gmx.net Mon Aug 27 21:59:56 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 27 Aug 2007 20:59:56 +0100 Subject: [Distutils] Uninstall In-Reply-To: <525f23e80708270311v4c3b93c2rc9e4e6fe59622984@mail.gmail.com> References: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> <525f23e80708270311v4c3b93c2rc9e4e6fe59622984@mail.gmail.com> Message-ID: Alexander Michael wrote: > On 8/26/07, Martin Aspeli wrote: >> Martin Aspeli wrote: >> I think it'd be a huge improvement if easy_install had a '--remove' >> option or something like that. Even if it just did this exact operation >> under the hood, it'd be a lot more intuitive. >> >> It'd be better if there was a way to remove the scripts and other egg >> files as well. I don't think people can be expected to realise where >> eggs are getting installed (i.e. site-packages/foo.egg, and maybe >> scripts/foobar and whatever else) and find and remove these files. >> >> Would easy_install in theory have enough context to work out what files >> was installed for an egg in the first place? > > As far as I was able to determine, the logic can be recreated, but I > couldn't find a record of the user chosen script directory (chosen > w/easy_install --script-dir). You could search on PATH, assuming the > user has put the script-dir on their PATH, but this starts to get > fragile (you might want to look in the file to be sure it is something > easy_install created, etc.). Maybe easy_install should also record these things during installation? > You might also run into some issues with projects that have been given > egg-info, but not installed by easy_install, not sure on this one. > > The route I chose to manage uninstalls is to use the --record option > to drive an uninstall script. Also fragile because I need to run > easy_install my way to make sure the record is generated, but > otherwise feels safer to me. > > I heartily second the need to do this at a more fundamental level. > Even if it is just unsinstalling without regard to dependencies. > Ultimately, it would be nice to "rotate" the installs, keep just the > latest, remove a project and all its dependencies that aren't > dependent on anohter installed project, etc., etc. But the first step > is being able to install a single project w/o dependencies. I'm sure there are lots of use cases here, and I understand it gets tricky with dependencies, but at the most basic level, I've seen several people hose their Python installs or workingenvs with incompatible eggs that they then try to remove. Martin -- Acquisition is a jealous mistress From lxander.m at gmail.com Mon Aug 27 23:51:52 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Mon, 27 Aug 2007 17:51:52 -0400 Subject: [Distutils] Uninstall In-Reply-To: References: <20070826234351.6BDDD3A4092@sparrow.telecommunity.com> <525f23e80708270311v4c3b93c2rc9e4e6fe59622984@mail.gmail.com> Message-ID: <525f23e80708271451g5da659b3p3812d99e9c767f49@mail.gmail.com> On 8/27/07, Martin Aspeli wrote: > Maybe easy_install should also record these things during installation? Personally, I think that would be helpful. I can think of three candidate places that might make sense: 1) right next to the egg as a pkg-1.0-py2.4.egg-record or something... 2) inside the installed egg as part of the egg-info as an installed_files.txt or some such... 3) in a "receipts" directory inside the install-dir The record would only be needed to allow uninstalling (so the std lib wouldn't need it). > I'm sure there are lots of use cases here, and I understand it gets > tricky with dependencies, but at the most basic level, I've seen several > people hose their Python installs or workingenvs with incompatible eggs > that they then try to remove. Yep. Especially on the bleeding edge of development... From lists at zopyx.com Tue Aug 28 15:17:17 2007 From: lists at zopyx.com (Andreas Jung) Date: Tue, 28 Aug 2007 15:17:17 +0200 Subject: [Distutils] Obtaining the Egg version from the Python code? Message-ID: <3765A45C6289C76BEE649A8F@suxmac2.local> Is there a way to obtain the version info of an egg from with Python code stored with the egg? Usecase: an egg contains several commandline utils (configured and installed using entry point). A dedicated --version option should display the overall version of the egg. Tnx, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070828/3f37a9a3/attachment.pgp From pje at telecommunity.com Tue Aug 28 17:30:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 28 Aug 2007 11:30:02 -0400 Subject: [Distutils] Obtaining the Egg version from the Python code? In-Reply-To: <3765A45C6289C76BEE649A8F@suxmac2.local> References: <3765A45C6289C76BEE649A8F@suxmac2.local> Message-ID: <20070828152741.78AC13A4072@sparrow.telecommunity.com> At 03:17 PM 8/28/2007 +0200, Andreas Jung wrote: >Is there a way to obtain the version info of an egg from with Python code >stored with the egg? > >Usecase: an egg contains several commandline utils (configured and installed >using entry point). A dedicated --version option should display the overall >version of the egg. pkg_resources.require('MyProjectName')[0].version From ivazqueznet at gmail.com Wed Aug 29 07:14:54 2007 From: ivazqueznet at gmail.com (Ignacio Vazquez-Abrams) Date: Wed, 29 Aug 2007 01:14:54 -0400 Subject: [Distutils] Disabling --single-version-externally-managed Message-ID: <1188364494.1528.8.camel@ignacio.lan> I have some packages that I'm using the following with: %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT I now need to install multiple versions of one of the packages and so need to disable the implicit --single-version-externally-managed. How can I do so and still use setup.py install? -- Ignacio Vazquez-Abrams -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070829/edc42fc0/attachment.pgp From pje at telecommunity.com Wed Aug 29 18:42:58 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Aug 2007 12:42:58 -0400 Subject: [Distutils] Disabling --single-version-externally-managed In-Reply-To: <1188364494.1528.8.camel@ignacio.lan> References: <1188364494.1528.8.camel@ignacio.lan> Message-ID: <20070829164031.24C963A4072@sparrow.telecommunity.com> At 01:14 AM 8/29/2007 -0400, Ignacio Vazquez-Abrams wrote: >I have some packages that I'm using the following with: > >%{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT > >I now need to install multiple versions of one of the packages and so >need to disable the implicit --single-version-externally-managed. How >can I do so and still use setup.py install? You can't. You can use something like this, though, to make an RPM that installs a non-active egg: setup.py easy_install --prefix=$RPM_BUILD_ROOT/usr -mx . The -mx means not to install a .pth or scripts, and the --prefix will set the base installation directory. The '.' means install the package whose source is in the current directory. From support at oeko.net Wed Aug 29 10:37:46 2007 From: support at oeko.net (Toni Mueller) Date: Wed, 29 Aug 2007 10:37:46 +0200 Subject: [Distutils] my wishlist for easy_install Message-ID: <20070829083746.20641.qmail@oak.oeko.net> Hi, I'd really like to see these two features in easy_install which I can't find right now: 1 - coexist with the operating system/recognize packages which are installed via operating system means 2 - remove packages Currently, when I want to easy_install a package, it starts downloading all sorts of stuff from PYPI although I have already similar (the same - maybe other versions) packages in my system which I installed via the operating system's package manager. I'd really like to not have to manually inspect a prospective package to then manually download and install all dependencies via the package manager, and then to proceed to easy_install -N the offending package, but instead have this process automated. I'd also be able to say something like easy_install -r package to remove such a package, but only if I installed it via easy_install in the first place, leaving operating system installed packages alone. I'm mostly using easy_install on Debian/Linux (Etch), but other platforms (OpenBSD, maybe Solaris) are relevant for me, too. Thanks for listening! Best, --Toni++