Mailman 3 July 2007 - Distutils-SIG

Re: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index.
by Phillip J. Eby 26 Jul '07

26 Jul '07

At 06:30 PM 7/24/2007 -0400, Noah Gift wrote: >Does this mean it is also easy to point to another local repository >that is available via NFS? I guess a local http mirror would work >just as well, if you told the opener about it. You don't even need to do that. See the -i/--index-url option to easy_install, for how to specify a specific package index URL to be used in place of PyPI. >This seems like a good way to instruct a sysadmin on how to setup a >local customized infrastructure! See also: http://peak.telecommunity.com/DevCenter/EasyInstall#creating-your-own-packa… http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

2 1

DistUtils does not install
by Rajeev Nair 26 Jul '07

26 Jul '07

Hi Folks, I am a complete newbie to python and its modules. I have got the following error upon installing DistUtils modules Python setup.py install Traceback (most recent call last): File "setup.py", line 30, in ? packages = ['distutils', 'distutils.command'], File "/usr/local/Distutils-1.0.2/distutils/core.py", line 101, in setup _setup_distribution = dist = klass(attrs) File "/usr/local/Distutils-1.0.2/distutils/dist.py", line 130, in __init__ setattr(self, method_name, getattr(self.metadata, method_name)) AttributeError: DistributionMetadata instance has no attribute 'get___doc__' What am I missing here ? Tried Installing some python-devel modules too but they just wouldn't install complaining about the python version to be 2.4. I have installed the same version of python though. The OS is Suse x86_64 GNU/Linux Please help. Regards, Rajeev

2 2

Regeneration of source manifest
by Arve Knudsen 25 Jul '07

25 Jul '07

Hello As far as I can tell setuptools does not always regenerate the source manifest when MANIFEST.in changes, contrary to what the documentation says. That is, if I add a directive to exclude files with the .txt extension to MANIFEST.in (after having previously including them), and run the sdist command, such files are still included. I have to remove the .egg-info for things to work as expected. This must clearly be a bug? Regards, Arve Knudsen

1 1

Questions regarding the compilation of a non-extension shared library
by James Dominy 25 Jul '07

25 Jul '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I'm have a shared library that I would like to distribute along with my python package, written in pure C. It requires the correct installation of other software at linktime, and at runtime. The package a ctypes wrapper for SUNDIALS (a mathematical C library). I would normally simply compile my shared library on the command line as follows - --- snip: shell output 1 --- $ gcc -c -o sundials_core_aux.o sundials_core_aux.c $ gcc --shared -fPIC -Wl,-solibsundials_core_aux.so.1 -o libsundials_core_aux.so.1.0.0 sundials_core_aux.o $ /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/../../../../i686-pc-linux-gnu/bin/ld: warning: creating a DT_TEXTREL in object. - --- snip --- To get rid of the DT_TEXTREL warning, I simply add -fPIC to compilation step. - --- snip: shell output 2 --- $ gcc --shared -fPIC -c -o sundials_core_aux.o $ gcc --shared -fPIC -Wl,-solibsundials_core_aux.so.1 -o libsundials_core_aux.so.1.0.0 sundials_core_aux.c $ - --- snip --- This produces no errors or warnings. Obviously this is a non-portable solution, and I'm trying to use distutils to compile the library. But when I do, using - --- snip: setup.py --- compiler = ccompiler.new_compiler() compiler.compile(['sundials_core_aux.c']) compiler.link_shared_lib(['src/sundials_core_aux.o'], compiler.library_filename('sundials_core_aux', 'shared'), 'dist') - --- snip --- The DT_TEXTREL warning appears. I've tried any number of combinations of extra_preargs and extra_postargs using '-fPIC' and '-Wl,-so...', to no avail, and equally, these are not portable arguments. Should the warning just be ignored, or is there a way I can get rid of it? Thanks, James -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) iD8DBQFGp2nYolkHbC41Z6URAguzAJ9GWnXSVsHl6T9M126jv5e3N6hxcACguPPv aCJiGxN3M8TVJ84oYYBsc7U= =I2Vn -----END PGP SIGNATURE-----

2 1

We need to make a decision wrt distribution names
by Jim Fulton 25 Jul '07

25 Jul '07

Obviously, we are having a debate about what forms distribution names can take. I think we need a decision. Does anyone know if there are existing rules for package names? I can't find them if there are. Up until now, I think we've been in somewhat of a prototyping mode, but I think it's time to move beyond that. I strongly suggest that we need an official specification that says: - what's a legal package name and - what the equivalence rules for package names are. Whatever we decide needs to be well supported by setuptools and PyPI. I can live with whatever we decide as long as we decide something and make sure it is well communicated and implemented. In particular, I could live with the equivalence rules that setuptools uses if they are documented and if they are supported correctly and efficiently by the index (including mirrors). IMO, a decision is extremely important. If we can't reach consensus, then we need to call in the BDFL. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

4 11

entry points
by Barry Warsaw 24 Jul '07

24 Jul '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just a quick question as I'm starting to play with this feature. Why do entry point values use a colon to separate the module from the callable? Couldn't you just as easily use a "normal" Python dotted path, the last component of which would of course be the callable? E.g. instead of setup( # ... entry_points = {'blogtool.parsers': '.rst = some_module:SomeClass'} ) use setup( # ... entry_points = {'blogtool.parsers': '.rst = some_module.SomeClass'} ) That would seem a bit more Pythonic. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRqZtZnEjvBPtnXfVAQIOHwP6AgX58dLBIzqvI7wo+z9MOIXm/rUIDikX j6Zc5S/97Fc9n3qjRlKf/fTT229oQVSIsPJgxvfXxWKYS5Qlk+YqhoACI9BuTbrm KshRSiPzFPy2eqjk287sHmrpq1MJWwrccjF4lSROn6VDcojlW/t2p6FY/+Gvj3au CPhRo/Amr+M= =8qj5 -----END PGP SIGNATURE-----

2 2

Re: [Distutils] [Catalog-sig] Prototype setuptools-specific PyPI index.
by Phillip J. Eby 24 Jul '07

24 Jul '07

At 04:37 PM 7/24/2007 -0400, Noah Gift wrote: >The basic algorithm is that a local index of PyPi could be kept in one >file. If an incorrect search was made, the first action to occur >would be to check if the local file was the same as the file on the >server. If not, it would sync the changes with svn. Then >easy_install would try to do lookups against the local file to find a >match. Note that there are a lot of ways you can implement something like this without even involving me on the client or Martin on the server. For example, setuptools.package_index uses urllib2 for all its URL access, so installing an "opener" that does caching before invoking easy_install is possible. You can also subclass the easy_install command class and the PackageIndex class, or tell the easy_install command class to use a different PackageIndex implementation. In the long run, I'd like to add some entry points to allow people to extend the search mechanism in such ways, but for now you can certainly hack subclasses easily enough and make your own alternative commands, as Jim has done for integrating zc.buildout with setuptools.

2 1

We need to make a decision wrt distribution names (second try)
by Jim Fulton 24 Jul '07

24 Jul '07

Obviously, we are having a debate about what forms distribution names can take. I think we need a decision. Does anyone know if there are existing rules for distribution names? I can't find them if there are. Up until now, I think we've been in somewhat of a prototyping mode, but I think it's time to move beyond that. I strongly suggest that we need an official specification that says: - what's a legal distribution name and - what the equivalence rules for distribution names are. Whatever we decide needs to be well supported by setuptools and PyPI. I can live with whatever we decide as long as we decide something and make sure it is well communicated and implemented. In particular, I could live with the equivalence rules that setuptools uses if they are documented and if they are supported correctly and efficiently by the index (including mirrors). IMO, a decision is extremely important. If we can't reach consensus, then we need to call in the BDFL. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

4 7

Prototype setuptools-specific PyPI index.
by Jim Fulton 24 Jul '07

24 Jul '07

Over the past few months, we've struggled quite a bit with Python Package Index (PyPI) performance and stability. Thanks to the heroic efforts of Martin v. Löwis and others, performance and especially stability have improved quite a bit. Martin has demonstrated that, at least when running well, PyPI seems to answer most requests on the order of 7 miliseconds (around 150 requests per second) internally. That's not bad. Unfortunately for users, actual times can be quite a bit longer. For me at work, request take around 300 milliseconds. For Martin, they seem to take somewhat longer. 300 milliseconds isn't so bad for a request or two, however, easy install can easily make 10s or even hundreds of requests to satisfy a user request for a package. zc.buildout, when verifying that a large system with many tens of packages has the most up to date versions of each package can easily make thousands of requests. Why do setuptools and buildout make so many requests? If a package exposes more than one release, then setuptools checks the package's main PyPI page and the pages for each release. We need to be able to easily use older releases, so we can't hide old releases. Typical projects of ours have many old releases exposed. If setuptools was more clever in the way it searched PyPI, but it would still have to make a minimum of 2 requests per package for packages with multiple versions exposed. Another potential issue is that PyPI pages can be large. I've found it convenient to use PyPI package pages as the home page for many of my projects. I like to include package documentation in my project pages. Perhaps this is an abuse of PyPI, but it is very convenient for me and no one has complained. :) The zc.buildout pages are around 200K. That's a fair bit of data for setuptools to download and scan for download URLs. In the course of this discussion, I've realized that it doesn't make sense for setuptools to use the same interface that humans use. setuptools doesn't need to see all of the data that is useful to humans. Similarly, humans generally don't need to see all of the historical releases for a project. I suggested a simple page format designed just for setuptools. An alternative would be an xmlrpc API. I prefer pages because I think that, over time, the amount of requests from automated tools like easy_install and zc.buildout will increase substantially and ultimately, will overwhelm dynamic servers, even ones like PyPI that are reasonably fast. I also think that a simple static collection of pages will be easier to mirror and I think some number of geographic mirrors is likely to help some people. I promised to prototype the format I suggested. I've created and experimental prototype setuptools-specific package index at http://download.zope.org/ppix Going to that page gives brief instructions for using it with easy_install and zc.buildout. To see an individual package page, add the package name to the URL, as in: http://download.zope.org/ppix/setuptools/ A few things to note about this: - I don't expose a long package list at http://download.zope.org/ ppix/. The long package list would be expensive to download and supports a use case that I consider to be of negative value, which is installing packages with case-insensitive package names, I think it is important for humans to be able to search for packages using case- insensitive search terms, but I think that, after identifying a package, precise package names should be used. I think it is especially important that precise package names be used in package requirements. - There is a single page per package. This can greatly reduce the number of requests. Packages that store all of their distributions in PyPI and that don't have off-site home pages or download URLs can be scanned with a single request. Note that I excluded home page and download URLs that pointed back to the packages PyPI page, as that wouldn't provide any new information to setuptools. - Download URLs for *hidden* packages are included. Humans don't need to see old revisions, but setuptools-based tools do. If we used an index like this for setuptools, we could stop unhiding old releases when we created new releases in PyPI. This would make PyPI more useful to humans and less of a pain for developers. - Download URLs are the same as they are in PyPI. Using this new index, distributions are still downloaded from PyPI, so the index doesn't affect PyPI download statistics. To see the impact of this, it's interesting to look at installing zc.buildout using easy_install from PyPI and from the experimental index: Installing using PyPI looks like this: (env)jim@ds9:~/tmp$ time easy_install zc.buildout Searching for zc.buildout Reading http://cheeseshop.python.org/pypi/zc.buildout/ Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b19 Reading http://svn.zope.org/zc.buildout Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b22 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b23 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b20 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b21 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b26 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b27 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b24 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b25 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b28 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b17 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b16 Reading http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b18 Best match: zc.buildout 1.0.0b28 Downloading http://cheeseshop.python.org/packages/2.5/z/ zc.buildout/zc.buildout-1.0.0b28- py2.5.egg#md5=4e37e53f010ed7984555a029732f479d Processing zc.buildout-1.0.0b28-py2.5.egg creating /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Extracting zc.buildout-1.0.0b28-py2.5.egg to /home/jim/tmp/env/lib/ python2.5 Adding zc.buildout 1.0.0b28 to easy-install.pth file Installing buildout script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Processing dependencies for zc.buildout Searching for setuptools==0.6c6 Best match: setuptools 0.6c6 Processing setuptools-0.6c6-py2.5.egg Adding setuptools 0.6c6 to easy-install.pth file Installing easy_install script to /home/jim/tmp/env/bin/ Installing easy_install-2.5 script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/setuptools-0.6c6-py2.5.egg Processing dependencies for setuptools==0.6c6 Finished processing dependencies for setuptools==0.6c6 Finished installing setuptools==0.6c6 Finished processing dependencies for zc.buildout Finished installing zc.buildout real 0m31.360s user 0m1.136s sys 0m0.060s Note the large number of pages read. Here I was installing a single package with one dependency, setuptools, that was already installed. Let's look at this again using the experimental index: (env)jim@ds9:~/tmp$ time easy_install -i http://download.zope.org/ ppix zc.buildout Searching for zc.buildout Reading http://download.zope.org/ppix/zc.buildout/ Best match: zc.buildout 1.0.0b28 Downloading http://cheeseshop.python.org/packages/2.5/z/ zc.buildout/zc.buildout-1.0.0b28- py2.5.egg#md5=4e37e53f010ed7984555a029732f479d Processing zc.buildout-1.0.0b28-py2.5.egg creating /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Extracting zc.buildout-1.0.0b28-py2.5.egg to /home/jim/tmp/env/lib/ python2.5 Adding zc.buildout 1.0.0b28 to easy-install.pth file Installing buildout script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/zc.buildout-1.0.0b28- py2.5.egg Processing dependencies for zc.buildout Searching for setuptools==0.6c6 Best match: setuptools 0.6c6 Processing setuptools-0.6c6-py2.5.egg Adding setuptools 0.6c6 to easy-install.pth file Installing easy_install script to /home/jim/tmp/env/bin/ Installing easy_install-2.5 script to /home/jim/tmp/env/bin/ Installed /home/jim/tmp/env/lib/python2.5/setuptools-0.6c6-py2.5.egg Processing dependencies for setuptools==0.6c6 Finished processing dependencies for setuptools==0.6c6 Finished installing setuptools==0.6c6 Finished processing dependencies for zc.buildout Finished installing zc.buildout real 0m7.006s user 0m0.244s sys 0m0.040s Note: - We made far fewer requests with the new index - Most of the time in the second example was spent actually downloading the buildout distribution. Most of the time in the first example was spent reading the index. - I used workingenv to create clean environments for each of the examples above. WRT zc.buildout, refreshing a buildout with just ZODB installed in it takes about 45 seconds for me using PyPI and about 5 seconds using the experimental index. Some of the speed improvements is due to the fact that the experimental index is much closer to me (on the net) than PyPI. ATM, requests to PyPI take *me* around 500 milliseconds, while requests to the experimental index are taking between 100 and 300 milliseconds. (I'm at home and this seems to be somewhat variable.) Most of the speed improvements are from reducing the number of requests. I'm polling PyPI once a minute to get and apply updates. Thanks to the new XML-RPC method that Martin added, this is very efficient to do. I encourage people to check this out and even try using it with easy_install and especially buildout. AFAIK, aside from being much faster and showing download files for hidden releases it is completely equivalent to PyPI for setuptools use. My intension is to keep this experimental index going and up to date for the foreseeable future and plan to use it for all my work. My primary goal is to prototype the new index format. If this seems useful, then I think that www.python.org should expose an index in this format to setuptools, either at a different URL or by satisfying setuptools requests from the index based on client information. I'd love to see this index populated via a baking mechanism that updates package pages when they change, rather than through polling as I'm doing. There would be some benefit to having geographic mirrors. I suspect that having such mirrors available would improve performance further, at least for some folks. It might also be useful to have some mirrors for redundancy purposes. Note though that what I'm doing is mirroring the only index data. I'm not mirroring distributions. Of course, I'd be happy to make my software available. (It already is via our subversion repository.) I hope this effort spurs useful discussion and progress. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

15 87

Problem with bdist_wininst option (trying to distribute a COM server, on Windows XP)
by Vinke, Johan 24 Jul '07

24 Jul '07

Preface: If this ends up being a real "Newbie" question, my apologies to all. But I have read "Distributing Python Modules" by Greg Ward, May 19, 2004, particularly Chapter 6 - Creating Built Distributions - and 6.3 - Creating Windows Installer. And I still can't see what I'm doing wrong. I'm having trouble using distutils with the bdist_wininst option. I've been able to create a COM server and package it in a .DLL, but I can't create an "installer" as a .exe My goal is to create a .exe that can be run on a client machine that does NOT have the standard Python environment, and this .exe would install all the modules?/.dll's/.exe's that are necessary for the COM server to run on the client machine. When I activate the "Run" command from within the PythonWin window and select the program setup_NaComponent_exe.py (attached), it nicely creates build and dist sub-directories. In the dist directory, there is a .exe and a .dll file: NaComponent.dll NaComponent.exe The interactive window has lots of informational messages such as: *** copy extensions *** . . *** copy dlls *** . . *** binary dependencies *** . . But the .exe and .dll created from the above (without the bdist_wininst argument) cannot be simply installed on a client machine and act as a COM server. (I tried registering the .dll with this command: regsvr32 /s \\Eng\prod\Group\TIS\Dev\Mfg\M3_New_PipeShop_COM\Current\Application\dis t\NaComponent.dll but that did not work. Next, I ran setup_NaComponent_exe.py, but I added the argument: bdist_wininst The messages returned in the interactive window were: >>> running bdist_wininst running build installing to build\bdist.win32\wininst >>> There was an error: ....\setup_NaComponent_exe.py' returned exit code error: build\bdist.win32\wininst: No such fi So I created the directory build\bdist.win32\wininst, reran the setup program, and got this: >>> running bdist_wininst running build installing to build\bdist.win32\wininst creating 'c:\docume~1\n50519\locals~1\temp\tmpaqvrsy.zip' and adding '.' to it removing 'build\bdist.win32\wininst' (and everything under it) >>> >>> But there was no zip file tmpaqvrsy.zip in the above-mentioned directory, and the wininst directory was removed (as described above). Can someone tell me what I'm doing wrong? Johan Vinke Nassco San Diego, CA _________________________________________________________________ Version info: Python Version: 2.3.3 PythonWin Version: win32all build 163 1st info line in PythonWin: PythonWin 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32.

2 1