What's the best way to extend distutils with "custom steps" like a call
to an IDL compiler or an M4 pre-process? I could just put all of the
code in my setup.py but it seems very inelegant and not necessarily
always effective (i.e. what if I have to do a post-process between the
build and install step?) Is there a technique I can use?
--
Vote for Your Favorite Python & Perl Programming
Accomplishments in the first Active Awards!
http://www.ActiveState.com/Awards
> On Wed, Feb 28, 2001 at 10:13:51AM +0100, Jack Jansen wrote:
> >Well, one of the things I'd like to do is integrate all this in MacPython IDE,
> >and the same probably holds for PythonWin and Idle. Then the "what do you
> >have" is pretty useful for building menus and such.
>
> I anticipate having a web-based interface to the database as well. I'm
> sure that we can get the CGI to let the user do that sort of search [...]
Right, but an IDE can do better. For instance, if an import fails it can offer
to do a lookup on the packages site to see whether such a package is
available. On the other hand: it could also do so by firing up your browser
with a carefully constructed URL. Ah well...
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
From: Sean Reifschneider [mailto:jafo@tummy.com]
> On Tue, Feb 27, 2001 at 09:30:13AM -0700, Evelyn Mitchell wrote:
> >But it will also discover and resolve dependences in your perl
> >site-packages, and automatically fetch them from your closest
> >CPAN archive.
>
> Not according to my tests the night before last. I did a test CPAN
> install of "News::Newsrc", which failed because the "make test" was
> failing. I then installed the "Set::BitSet" (? Something like that)
> module and tried News::Newsrc again and it worked...
>
> Maybe this was just a fluke and News::Newsrc is the exception and/or
> isn't used enough that people have gotten the prereqs right yet. If
> anyone knows for sure, I'm curious.
There are basically a number of aspects to "CPAN", which need separating
out.
MakeMaker
---------
This is a perl module which implements a build process. You write a
Makefile.PL, which calls MakeMaker defining the module's structure and
metadata. Thanks to MakeMaker, the process for building a Perl module is
(nearly always) simply
perl Makefile.PL
make
make test <-- optional, but pretty much standard - runs module unit
tests
make install <-- installs the module
This is, in both concept and functionality, almost identical to Distutils.
There are some areas in which distutils is better (building of
platform-specific installers, for instance) and some where MakeMaker is
better (it hooks into a very standard structure for modules, generated by
thye h2xs program, which practically forces module authors to write test
suites and documentation in a standard format), but these are details.
We can consider this covered. (Although the distutils-sig could still
usefully learn from MakeMaker).
The system of FTP sites and mirrors
-----------------------------------
Frankly, this is nothing special. It has some nice features (automated
uploads for module authors, plus quite nice indexing and server multiplexing
features), but it isn't rocket science as far as I know. We could quite
happily start with a simple FTP site for this (that's what CPAN did - the
mirroring came later as popularity grew).
CPAN.pm
-------
This is a Perl module, which automates the process of downloading and
installing modules. I don't use this personally, for a number of reasons.
Frankly, I find that manual downloading and running the 4 lines noted above
is fine.
It relies for things like dependency tracking on metadata added into the
Makefile.PL which is not necessary for the straight 4-line build above. As
such, the level to which that metadata is added by module authors is
variable (to be polite). In practice, I wouldn't rely on it - accurate
dependency data seems to be the exception rather than the rule.
I *don't* regard CPAN.pm to be important to the overall CPAN "phenomenon".
But some people like it. Writing something "at least as good as" CPAN.pm
shouldn't be hard in Python - not least because the standard library is rich
enough that things like FTP client support is available out of the box
(whereas CPAN.pm won't work until you manually install libnet, and possibly
some other modules, I forget which...)
But writing a "perfect" utility for automated download-and-install, with
dependency tracking, etc etc, is going to be VERY HARD. Don't get misled -
Perl doesn't have such a beast. And We won't even have what Perl has if we
focus on perfection rather than practicality.
The h2xs program
----------------
This is VERY important. The idea is that when you write a Perl module,
either pure perl or a C (XS) extension, you run h2xs first, to generate a
template build directory. It automatically includes
* The perl module, with some basic template code and embedded POD
documentation
* The XS extension, with template code (if requested)
* A Makefile.PL shell
* A basic test script - all it does is test that the module loads,
but it includes a placeholder for your own tests
Essentially, h2xs forces a standard structure on all Perl modules. This is
important for Perl, where modules have to conform to some standards in order
to work at all. However, it brings HUGE benefits in standardisation of all
the "other" parts of the process (documentation, tests, etc).
Python is at a disadvantage here, precisely because writing a Python module
involves so little in the way of a specific structure. So people will likely
rebel against having a structure "imposed"...
A social structure
------------------
This is a bit of a chicken and egg issue, but Perl developers expect to
write modules using h2xs and MakeMaker, they expect to write tests (even if
they are minimal), they expect to fill in the sections in the POD
documentation, and they expect to submit their modules to CPAN. So this all
"just works".
Interestingly, developers probably don't "expect" to have to include
dependency information, and hence many don't - resulting in the problems you
hit. But then again, Perl users don't "expect" to be totally shielded from
dependency issues.
Python is VERY far behind here. This is a maturity issue - distutils is
still (relatively) new, and so there are LOTS of packages which don't come
with a setup.py yet. Often, adding one isn't hard, but it is still to
happen. And when you are distributing a pure python module, as a single .py
file, it's hard to see the benefit of changing that into a .tar.gz file
containing the module, plus a setup.py. (Once you start adding test suites
and documentation, the point of the whole archive bit is clearer, but we're
not even close to that stage yet).
Things are looking better with Python 2.1, though. Included with 2.1, it
looks like there will be a standard unit testing module, and a new "pydoc"
package, which will extract documentation from module docstrings. So the
infrastructure will be there, it just becomes a case of getting developers
to use it consistently. Distutils can help with this, by "just working" if
people follow a standard structure.
Sorry, this wasn't meant to get so long...
Paul.
> I'm not sure why the "what do you have" question is needed. The "send me
> that(mandrake.rpm)" interaction is what we want.
Well, one of the things I'd like to do is integrate all this in MacPython IDE,
and the same probably holds for PythonWin and Idle. Then the "what do you
have" is pretty useful for building menus and such.
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
hi !
recently i experimented with the very bad native c++ support of python &
distutils,
but 'found' a very good library which adds exactly this to python
(distutils).
writing extensions in c is fully supported by the distutils.
not a single error or warning on compilation. very nice!
but compiling and extension-builing of c++ modules is not so extremely easy.
(small probs with library-path, compileroptions, linkeroptions, no special
distutils 'Extension')
boring.
that's why i'd want to propose to include and support the boost python
library
especialy into the python documentation, examples,
and to special support them in distutils.
so people dont have to go and search for their own, and all using
different libs and inventing the wheel twice.
(same thing with py2exe)
proposal:
"""
1) include the boost-python c++ extension library into the distutils to get
the opensource power concentrated on this fine library. finaly making it the
'standart'.
2) add some more power to the build process of the distutils, by
automaticaly considering the systems 'include' variable.
3) add system independend compile-options to the distutils.
+RTTI -RTTI
[+Force_C -Force_C ]
[+Force_C++ -Force_C++]
...
(because boost needs >extra_compile_args=["/GR"]< and it would be great
to have
this determinated automaticaly (regex-search for "dynamic_cast" in the
library source).
this would hit in "extension_class.hpp" for boost)
4) most important: document the use of boost in the python documentation, so
writing
c++ extension becomes the most natural thing one comes in mind to extend
python with.
"""
maybe you remember me, some time ago i posted a question, how to extend
python with c++ code.
i got a hint about swig. (greetings to thomas heller!)
so i got myself going and started to explore the possibilities, beginning
with swig.
my findings were basicaly:
swig (shadowclasses-mode)
cXX (very limited)
zope-ExtensionClass
boost-ExtensionClass
i compared the tools, and found cxx and swig unusable for large projects.
(containing overloaded operators, templates..)
wxwindows is not using this powerfull extended features, and so swig does a
good job this case.
dont argue about this.
and yes, wrapping for instance mesa with boost makes no sense. swig is
probably still the best for such tasks. but that's not the point here.
however, i found that the boost-ExtensionClass had the most advanced design,
which simplifies even the reference-counting and operator overloading for
you.
please visit the comparison-page of boost for pro and con of most existing
tools to make writing c++ extensions easy.
1)
the comparison page from boost says it all.
swig and cxx won't do. as we have an interpreted language,
we definitly dont want to use even another intermediate 'compiler'
to get and maintain our extensions.
+ of course everyone is using templates, overloaded operators,
and virtual functions this days. (great features, cant live without them).
zope is great, but boost is basicaly same, and seems to do more. (still
so?)
it's very easy to incorporate into the c++ code, so it will not
mess up python code in any way (wrappers, shaddow-classes, quirky calling
conventions)
and you need to compile your extension anyway, right?
so i believe, the python community needs a fully supported 'foolproof'
default way for POWERFULL c++ extensions,
which feel like python classes, once imported.
first this means: build of bosst-lib itself should be automated using
distutils "make_boost_lib(path)" which makes the lib, and automaticaly links
it to the extensions build subsequently using boost headers.
(you'll find a better sytax. it's just the idea.)
why? you are right, everyone CAN write this into a setup-script already,
but why was distutils introduced? -> your answer.
i tried, but gave up on compilerswitches. i used the included
msvc workspace to compile boost. uhh..
2)
from doc:
'os.confstr(name)' #Return string-valued system configuration values.
is documented in os. but works not for windows. (at least for me)
this command (?) is neccessary to implement a systemvariable lookup
for the "include" variable, people use to 'install' librarys.
system_incl_dirs = os.confstr("include")
system_incl_dirs += os.confstr("Include")
extension.py line 98 can become:
if include_dirs is None: include_dirs=[]
self.include_dirs = include_dirs + [system_incl_dirs]
now it would be easy for me to extend os for confstr,
but i did not found the body of this function (using a overall search)
and dont want to mess up something i just begin to understand
should i implement it?
this change opens the door for python extensions to all 'installed'
librarys (ok, some ideas for the linker?)
3)
noone wants to look up man gcc + msvc + bc5.5 for the exact syntax of a
special compilerswitch. (platform)
people have more important stuff to do.
4)
docs are the key to get features used.
======================================
documentation of boost - python lib is erm.. medium.
i would contribute a rewritten documentation of boost,
and self-made examples.
not the incomplete buggy examples given in the boost doc,
but compiling and working ones. (using modified distutils + a setup.py)
please tell me your opinion!
i think of c++ and python as the 2 languages of the future. (for general
programming tasks)
c++ parts interconnected by python code, which even binds other crap
erm.. code crash-proof (exceptions) to the whole package if realy neccesary.
easily maintainable by changing the script holding high quality, fast,
templated c++ libs and apps together. python is the dot which makes c++ perfect.
actualy i reorganize my c++ code (mostly commandline) to be controlled from
python (using boost) and give them a gui (wxPython (by dunn, thanks man!)).
this heavily uses:
c++ (+stl+templates+virtual_funcs+operators..)
python (laaaate binding)
boost (c++ python interconnection)
wxwindows (best crossplatform gui.)
wxpython (swig python-binding of the above)
py mods (ftp, url, regex, ...)
py2exe (commercial stuff.. need to keep my refrigerator filled by myself
:-(
i think this is a fairly robust design-path.
(thus deserving special support by python + distutils i believe.)
ok, that's all. at least for now :-)
no flames please.
ml
--
Sent through GMX FreeMail - http://www.gmx.net
In the past I have used the "freeze" method available in the tools section
of the Python distribution for distributing binaries to those without
Python. Just recently I started using DistUtils for compiling my C extension
but unfortunately that breaks the freezing process if the extension module
is going to be folded in to the resulting executable (rather than a
standalone shared library). Before I go off and try to make this work, I was
wondering if anyone knew if freezing was being planned for DistUtils. If so,
what can I do to help, and if not, what ought to be replacing it? I suspect
it shouldn't be too difficult to take the code that determines modules,
compile them and generate C files, then use the DistUtils modules to
actually compile the executable. But I don't want to do that if someone
already has or there is something better to use. Comments anyone?
Anthony
Unfortunately, this week has been pretty busy. I've been giving the idea
some thought again (it's been something I've wanted to work on for quite
a while), and just finally today had the chance to look at pythonsiphon.
I'm not really clear (from the code what) as to what it's job is to be.
It smells like it's wanting to be something that understands, given a
distutils package, how to install it.
My thoughts on that are that it's a job that's probably going to be
something subject largely to being handed off to appropriate scripts
based on the platform and some user input (for example, a user may
prefer to have packages they download installed in ~/lib/python1.5,
instead of /usr/lib/python1.5).
Ideally though, the tool should be able to deal with allowing the user
to select their preferred distribution media. I'd prefer to see an ix86
RedHat 7.0 RPM, a SRPM, and then would fall back to a distutils file or
tar file.
As far as the server side for allowing users to query packages,
dependencies, and locations...
I had actually hoped to pound out some code on this, but that just
didn't happen. I have built up a schema which I think encompasses
everything I see such a system wanting to track. Initial versions
probably won't track all that, but I believe they should be considered.
The schema is attached below.
A quick overview of the tables. I hate the name "items", but it's
the best I could come up with. That's what I'm calling the general
module (things that can be imported) or package (for example, programs
like Sketch -- why not have them in here as well) description. In
this case, let's use "sockserv". The URL would point to the main
location you can find "sockserv" module information.
"Packages" are actually meant to describe what you can download.
This database will track multiple versions, include a checksum
and information on what platform it's meant for (for binary
distributions, for example).
Mirrors are handled by a special table. The idea is that one should
be able to add mirrors without having to list all the items in that
particular mirror (again). New entries should be able to be listed
without having to list all the mirrors they might reside on. So,
I've come up with the idea that a mirror simply specifies a "prefix"
for the URL listed in the "locations" table.
To be honest, my plan is to have a special-case "null" mirror which
non-mirror members are listed under -- that would have an empty
prefix.
So, given a package, version, format, architecture, etc... The
"locations" table will produce list of all locations which that
file can be found at.
Note that there is a dependency table which lists that one "item"
relies on another "item".
As far as the network interface goes, at the current moment I'm thinking
that queries will be submitted to a web server CGI as a POST, and the
results will be returned as the body as text/plain... I'll probably use
netstrings (as discussed before) to encode things for transport.
So, any comments?
Sean
=========================
#ISSUES:
# This doesn't handle partial mirrors. Should it? How?
CREATE TABLE items (
id serial primary key,
lastUpdated datetime default 'now',
name text,
description text,
type text, # module, package
homepageURL text,
)
CREATE TABLE packages (
id serial primary key,
itemID int4,
version text,
md5sum text,
filesize text,
insertedOn datetime default 'now',
description text, # description about this specific file
platformName text,
platformVersion text,
architecture text,
packageFormat text,
)
# a collection of URLs that all contain the same set of files (within some
# precision). A mirror can be added without having to add all the URLs it
# contains. Not useful for handling partial-mirrors though.
CREATE TABLE mirrors (
id serial primary key,
name text,
baseURL text,
)
CREATE TABLE locations (
id serial primary key,
packageID int4,
mirrorID int4, # if found on a mirror, list it
URL text, # append this text to the mirror baseURL
)
CREATE TABLE depends (
id serial primary key,
itemID int4,
requiresItemID int4
)
CREATE TABLE users (
id serial primary key,
name text,
password text,
lastLogin datetime default NULL,
loginCookie text
)
CREATE TABLE maintainers (
userID int4,
type text, # package, item
piID int4
)
CREATE TABLE ranking (
id serial primary key,
rank int2,
itemID int4,
userID int4
date datetime default 'now',
)
--
Millions long for immortality who don't know what to do with themselves
on a rainy Sunday afternoon. -- Heinlein
Sean Reifschneider, Inimitably Superfluous <jafo(a)tummy.com>
tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python
