Distutils-SIG March 2001

distutils-sig@python.org

33 participants
57 discussions

by Paul Prescod

What's the best way to extend distutils with "custom steps" like a call to an IDL compiler or an M4 pre-process? I could just put all of the code in my setup.py but it seems very inelegant and not necessarily always effective (i.e. what if I have to do a post-process between the build and install step?) Is there a technique I can use? -- Vote for Your Favorite Python & Perl Programming Accomplishments in the first Active Awards! http://www.ActiveState.com/Awards

19 years, 6 months

Re: [Distutils] Re: CPAN functionality for python - requirements

by Jack Jansen

> On Wed, Feb 28, 2001 at 10:13:51AM +0100, Jack Jansen wrote: > >Well, one of the things I'd like to do is integrate all this in MacPython IDE, > >and the same probably holds for PythonWin and Idle. Then the "what do you > >have" is pretty useful for building menus and such. > > I anticipate having a web-based interface to the database as well. I'm > sure that we can get the CGI to let the user do that sort of search [...] Right, but an IDE can do better. For instance, if an import fails it can offer to do a lookup on the packages site to see whether such a package is available. On the other hand: it could also do so by firing up your browser with a carefully constructed URL. Ah well... -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

19 years, 6 months

RE: [Distutils] Re: CPAN functionality for python - requirements

by Moore, Paul

From: Sean Reifschneider [mailto:jafo@tummy.com] > On Tue, Feb 27, 2001 at 09:30:13AM -0700, Evelyn Mitchell wrote: > >But it will also discover and resolve dependences in your perl > >site-packages, and automatically fetch them from your closest > >CPAN archive. > > Not according to my tests the night before last. I did a test CPAN > install of "News::Newsrc", which failed because the "make test" was > failing. I then installed the "Set::BitSet" (? Something like that) > module and tried News::Newsrc again and it worked... > > Maybe this was just a fluke and News::Newsrc is the exception and/or > isn't used enough that people have gotten the prereqs right yet. If > anyone knows for sure, I'm curious. There are basically a number of aspects to "CPAN", which need separating out. MakeMaker --------- This is a perl module which implements a build process. You write a Makefile.PL, which calls MakeMaker defining the module's structure and metadata. Thanks to MakeMaker, the process for building a Perl module is (nearly always) simply perl Makefile.PL make make test <-- optional, but pretty much standard - runs module unit tests make install <-- installs the module This is, in both concept and functionality, almost identical to Distutils. There are some areas in which distutils is better (building of platform-specific installers, for instance) and some where MakeMaker is better (it hooks into a very standard structure for modules, generated by thye h2xs program, which practically forces module authors to write test suites and documentation in a standard format), but these are details. We can consider this covered. (Although the distutils-sig could still usefully learn from MakeMaker). The system of FTP sites and mirrors ----------------------------------- Frankly, this is nothing special. It has some nice features (automated uploads for module authors, plus quite nice indexing and server multiplexing features), but it isn't rocket science as far as I know. We could quite happily start with a simple FTP site for this (that's what CPAN did - the mirroring came later as popularity grew). CPAN.pm ------- This is a Perl module, which automates the process of downloading and installing modules. I don't use this personally, for a number of reasons. Frankly, I find that manual downloading and running the 4 lines noted above is fine. It relies for things like dependency tracking on metadata added into the Makefile.PL which is not necessary for the straight 4-line build above. As such, the level to which that metadata is added by module authors is variable (to be polite). In practice, I wouldn't rely on it - accurate dependency data seems to be the exception rather than the rule. I *don't* regard CPAN.pm to be important to the overall CPAN "phenomenon". But some people like it. Writing something "at least as good as" CPAN.pm shouldn't be hard in Python - not least because the standard library is rich enough that things like FTP client support is available out of the box (whereas CPAN.pm won't work until you manually install libnet, and possibly some other modules, I forget which...) But writing a "perfect" utility for automated download-and-install, with dependency tracking, etc etc, is going to be VERY HARD. Don't get misled - Perl doesn't have such a beast. And We won't even have what Perl has if we focus on perfection rather than practicality. The h2xs program ---------------- This is VERY important. The idea is that when you write a Perl module, either pure perl or a C (XS) extension, you run h2xs first, to generate a template build directory. It automatically includes * The perl module, with some basic template code and embedded POD documentation * The XS extension, with template code (if requested) * A Makefile.PL shell * A basic test script - all it does is test that the module loads, but it includes a placeholder for your own tests Essentially, h2xs forces a standard structure on all Perl modules. This is important for Perl, where modules have to conform to some standards in order to work at all. However, it brings HUGE benefits in standardisation of all the "other" parts of the process (documentation, tests, etc). Python is at a disadvantage here, precisely because writing a Python module involves so little in the way of a specific structure. So people will likely rebel against having a structure "imposed"... A social structure ------------------ This is a bit of a chicken and egg issue, but Perl developers expect to write modules using h2xs and MakeMaker, they expect to write tests (even if they are minimal), they expect to fill in the sections in the POD documentation, and they expect to submit their modules to CPAN. So this all "just works". Interestingly, developers probably don't "expect" to have to include dependency information, and hence many don't - resulting in the problems you hit. But then again, Perl users don't "expect" to be totally shielded from dependency issues. Python is VERY far behind here. This is a maturity issue - distutils is still (relatively) new, and so there are LOTS of packages which don't come with a setup.py yet. Often, adding one isn't hard, but it is still to happen. And when you are distributing a pure python module, as a single .py file, it's hard to see the benefit of changing that into a .tar.gz file containing the module, plus a setup.py. (Once you start adding test suites and documentation, the point of the whole archive bit is clearer, but we're not even close to that stage yet). Things are looking better with Python 2.1, though. Included with 2.1, it looks like there will be a standard unit testing module, and a new "pydoc" package, which will extract documentation from module docstrings. So the infrastructure will be there, it just becomes a case of getting developers to use it consistently. Distutils can help with this, by "just working" if people follow a standard structure. Sorry, this wasn't meant to get so long... Paul.

19 years, 6 months

Re: [Distutils] Re: CPAN functionality for python - requirements

by Jack Jansen

> I'm not sure why the "what do you have" question is needed. The "send me > that(mandrake.rpm)" interaction is what we want. Well, one of the things I'd like to do is integrate all this in MacPython IDE, and the same probably holds for PythonWin and Idle. Then the "what do you have" is pretty useful for building menus and such. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

19 years, 6 months

changes neccesary to natively support unlimited c++ extensions

by matthes markers

hi ! recently i experimented with the very bad native c++ support of python & distutils, but 'found' a very good library which adds exactly this to python (distutils). writing extensions in c is fully supported by the distutils. not a single error or warning on compilation. very nice! but compiling and extension-builing of c++ modules is not so extremely easy. (small probs with library-path, compileroptions, linkeroptions, no special distutils 'Extension') boring. that's why i'd want to propose to include and support the boost python library especialy into the python documentation, examples, and to special support them in distutils. so people dont have to go and search for their own, and all using different libs and inventing the wheel twice. (same thing with py2exe) proposal: """ 1) include the boost-python c++ extension library into the distutils to get the opensource power concentrated on this fine library. finaly making it the 'standart'. 2) add some more power to the build process of the distutils, by automaticaly considering the systems 'include' variable. 3) add system independend compile-options to the distutils. +RTTI -RTTI [+Force_C -Force_C ] [+Force_C++ -Force_C++] ... (because boost needs >extra_compile_args=["/GR"]< and it would be great to have this determinated automaticaly (regex-search for "dynamic_cast" in the library source). this would hit in "extension_class.hpp" for boost) 4) most important: document the use of boost in the python documentation, so writing c++ extension becomes the most natural thing one comes in mind to extend python with. """ maybe you remember me, some time ago i posted a question, how to extend python with c++ code. i got a hint about swig. (greetings to thomas heller!) so i got myself going and started to explore the possibilities, beginning with swig. my findings were basicaly: swig (shadowclasses-mode) cXX (very limited) zope-ExtensionClass boost-ExtensionClass i compared the tools, and found cxx and swig unusable for large projects. (containing overloaded operators, templates..) wxwindows is not using this powerfull extended features, and so swig does a good job this case. dont argue about this. and yes, wrapping for instance mesa with boost makes no sense. swig is probably still the best for such tasks. but that's not the point here. however, i found that the boost-ExtensionClass had the most advanced design, which simplifies even the reference-counting and operator overloading for you. please visit the comparison-page of boost for pro and con of most existing tools to make writing c++ extensions easy. 1) the comparison page from boost says it all. swig and cxx won't do. as we have an interpreted language, we definitly dont want to use even another intermediate 'compiler' to get and maintain our extensions. + of course everyone is using templates, overloaded operators, and virtual functions this days. (great features, cant live without them). zope is great, but boost is basicaly same, and seems to do more. (still so?) it's very easy to incorporate into the c++ code, so it will not mess up python code in any way (wrappers, shaddow-classes, quirky calling conventions) and you need to compile your extension anyway, right? so i believe, the python community needs a fully supported 'foolproof' default way for POWERFULL c++ extensions, which feel like python classes, once imported. first this means: build of bosst-lib itself should be automated using distutils "make_boost_lib(path)" which makes the lib, and automaticaly links it to the extensions build subsequently using boost headers. (you'll find a better sytax. it's just the idea.) why? you are right, everyone CAN write this into a setup-script already, but why was distutils introduced? -> your answer. i tried, but gave up on compilerswitches. i used the included msvc workspace to compile boost. uhh.. 2) from doc: 'os.confstr(name)' #Return string-valued system configuration values. is documented in os. but works not for windows. (at least for me) this command (?) is neccessary to implement a systemvariable lookup for the "include" variable, people use to 'install' librarys. system_incl_dirs = os.confstr("include") system_incl_dirs += os.confstr("Include") extension.py line 98 can become: if include_dirs is None: include_dirs=[] self.include_dirs = include_dirs + [system_incl_dirs] now it would be easy for me to extend os for confstr, but i did not found the body of this function (using a overall search) and dont want to mess up something i just begin to understand should i implement it? this change opens the door for python extensions to all 'installed' librarys (ok, some ideas for the linker?) 3) noone wants to look up man gcc + msvc + bc5.5 for the exact syntax of a special compilerswitch. (platform) people have more important stuff to do. 4) docs are the key to get features used. ====================================== documentation of boost - python lib is erm.. medium. i would contribute a rewritten documentation of boost, and self-made examples. not the incomplete buggy examples given in the boost doc, but compiling and working ones. (using modified distutils + a setup.py) please tell me your opinion! i think of c++ and python as the 2 languages of the future. (for general programming tasks) c++ parts interconnected by python code, which even binds other crap erm.. code crash-proof (exceptions) to the whole package if realy neccesary. easily maintainable by changing the script holding high quality, fast, templated c++ libs and apps together. python is the dot which makes c++ perfect. actualy i reorganize my c++ code (mostly commandline) to be controlled from python (using boost) and give them a gui (wxPython (by dunn, thanks man!)). this heavily uses: c++ (+stl+templates+virtual_funcs+operators..) python (laaaate binding) boost (c++ python interconnection) wxwindows (best crossplatform gui.) wxpython (swig python-binding of the above) py mods (ftp, url, regex, ...) py2exe (commercial stuff.. need to keep my refrigerator filled by myself :-( i think this is a fairly robust design-path. (thus deserving special support by python + distutils i believe.) ok, that's all. at least for now :-) no flames please. ml -- Sent through GMX FreeMail - http://www.gmx.net

19 years, 6 months

Question

by Anthony Tuininga

In the past I have used the "freeze" method available in the tools section of the Python distribution for distributing binaries to those without Python. Just recently I started using DistUtils for compiling my C extension but unfortunately that breaks the freezing process if the extension module is going to be folded in to the resulting executable (rather than a standalone shared library). Before I go off and try to make this work, I was wondering if anyone knew if freezing was being planned for DistUtils. If so, what can I do to help, and if not, what ought to be replacing it? I suspect it shouldn't be too difficult to take the code that determines modules, compile them and generate C files, then use the DistUtils modules to actually compile the executable. But I don't want to do that if someone already has or there is something better to use. Comments anyone? Anthony

19 years, 6 months

Re: CPAN functionality for python

by Sean Reifschneider

Unfortunately, this week has been pretty busy. I've been giving the idea some thought again (it's been something I've wanted to work on for quite a while), and just finally today had the chance to look at pythonsiphon. I'm not really clear (from the code what) as to what it's job is to be. It smells like it's wanting to be something that understands, given a distutils package, how to install it. My thoughts on that are that it's a job that's probably going to be something subject largely to being handed off to appropriate scripts based on the platform and some user input (for example, a user may prefer to have packages they download installed in ~/lib/python1.5, instead of /usr/lib/python1.5). Ideally though, the tool should be able to deal with allowing the user to select their preferred distribution media. I'd prefer to see an ix86 RedHat 7.0 RPM, a SRPM, and then would fall back to a distutils file or tar file. As far as the server side for allowing users to query packages, dependencies, and locations... I had actually hoped to pound out some code on this, but that just didn't happen. I have built up a schema which I think encompasses everything I see such a system wanting to track. Initial versions probably won't track all that, but I believe they should be considered. The schema is attached below. A quick overview of the tables. I hate the name "items", but it's the best I could come up with. That's what I'm calling the general module (things that can be imported) or package (for example, programs like Sketch -- why not have them in here as well) description. In this case, let's use "sockserv". The URL would point to the main location you can find "sockserv" module information. "Packages" are actually meant to describe what you can download. This database will track multiple versions, include a checksum and information on what platform it's meant for (for binary distributions, for example). Mirrors are handled by a special table. The idea is that one should be able to add mirrors without having to list all the items in that particular mirror (again). New entries should be able to be listed without having to list all the mirrors they might reside on. So, I've come up with the idea that a mirror simply specifies a "prefix" for the URL listed in the "locations" table. To be honest, my plan is to have a special-case "null" mirror which non-mirror members are listed under -- that would have an empty prefix. So, given a package, version, format, architecture, etc... The "locations" table will produce list of all locations which that file can be found at. Note that there is a dependency table which lists that one "item" relies on another "item". As far as the network interface goes, at the current moment I'm thinking that queries will be submitted to a web server CGI as a POST, and the results will be returned as the body as text/plain... I'll probably use netstrings (as discussed before) to encode things for transport. So, any comments? Sean ========================= #ISSUES: # This doesn't handle partial mirrors. Should it? How? CREATE TABLE items ( id serial primary key, lastUpdated datetime default 'now', name text, description text, type text, # module, package homepageURL text, ) CREATE TABLE packages ( id serial primary key, itemID int4, version text, md5sum text, filesize text, insertedOn datetime default 'now', description text, # description about this specific file platformName text, platformVersion text, architecture text, packageFormat text, ) # a collection of URLs that all contain the same set of files (within some # precision). A mirror can be added without having to add all the URLs it # contains. Not useful for handling partial-mirrors though. CREATE TABLE mirrors ( id serial primary key, name text, baseURL text, ) CREATE TABLE locations ( id serial primary key, packageID int4, mirrorID int4, # if found on a mirror, list it URL text, # append this text to the mirror baseURL ) CREATE TABLE depends ( id serial primary key, itemID int4, requiresItemID int4 ) CREATE TABLE users ( id serial primary key, name text, password text, lastLogin datetime default NULL, loginCookie text ) CREATE TABLE maintainers ( userID int4, type text, # package, item piID int4 ) CREATE TABLE ranking ( id serial primary key, rank int2, itemID int4, userID int4 date datetime default 'now', ) -- Millions long for immortality who don't know what to do with themselves on a rainy Sunday afternoon. -- Heinlein Sean Reifschneider, Inimitably Superfluous <jafo(a)tummy.com> tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python

19 years, 6 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG March 2001