Building Python on Cray T3E

[Mark Hadfield]

From: "Michael Hudson" <mwh at python.net>
> "Mark Hadfield" <m.hadfield at niwa.co.nz> writes:
>>   - As noted earlier in this thread, there is a C-function name
>>   clash in the _sre module, which can be easily fixed. However,
>>   recall that sre is one of two alternative regular-expression
>>   engines for the re module, the other engine being pre. It turns
>>   out that sre does not pass its tests so the path of least
>>   resistance is is to modify Lib/re.py to use the pre engine (the
>>   change is trivial) and comment out the _sre entry in
>>   Modules/Setup.
> 
> How does it fail?  Is it the "NoneType not callable" failure below.

Oops. The test_sre failure occurred because it couldn't load the
unicodedata module. I thought I had tried and failed to build
unicodedata, but in fact I hadn't tried. I did try, it builds OK, and
test_re succeeds. So cancel my suggestion above. DO patch & build
_sre.c, DON'T modify Lib/re.py. DO make sure you build unicodedata.c.

The "NoneType not callable" failure comes from pre.

>>     test test_re crashed -- exceptions.TypeError: 'NoneType' object
>>     is not callable
> 
> This is just weird.  Can you dig a little?

OK, just a little so far. It's not Cray-related. It occurs on other
platforms if you modify re.py to select the pre engine. I've seen the
same exception elsewhere when a function argument is missing. So I'm
guessing that the problem is that pre does not present exactly the
same interface as sre.

It's a moot point for me now that I have sre working, but is the pre
module still supported? If so should I enter this failure as a bug in
pre?

>>   - After building Python, make runs setup.py, which attempts
>>   (among other things) to build extensions not already built &
>>   linked statically. This is a futile but harmless exercise; the
>>   inevitable failures do not cause make to abort. I have seen
>>   elsewhere (in connection with Cygwin Python) a reference to a
>>   "--disable-shared" option to configure that I presume would
>>   inhibit this. However this is ignored by configure in Python
>>   2.2.1.
> 
> You can get it to not do this by just executing "make python" rather
> than "make all" or "make".  --disable-shared is a red herring in
> this context, I think (even in 2.3 where it exists).

Executing "make python" does indeed bypass attempts to build extension
modules for dynamic linking, but "make install" and "make test"
don't. My ad-hoc solution is to change the following in setup.py
(class PyBuildExt(build_ext), function build_extensions(self)):

        # Remove modules that are present on the disabled list
        self.extensions = [ext for ext in self.extensions
                           if ext.name not in disabled_module_list]

to this:

        self.extensions = []

>>   - Testing is a somewhat depressing experience. On the NIWA
>>   machine, a simple "make test" command fails to complete after 12
>>   hours and thrashes the swap space, with the side effect of
>>   inhibiting all network access for 10 seconds out of every 30.
> 
> !  Is this test_longexp, by any chance?

Not specifically that one. I think the problem is that running all the
tests in sequence without reloading the Python interpreter demands a
lot of memory. On our Compaq Unix machine it takes 67 MB and on the
Cray it takes 128 MB. This doesn't sound like an excessive amount but
Cray T3E's are not very smart about swapping and we seem to have some
sort of load-balancing problem that causes the single-CPU jobs to
flock to one processor.

>> My immediate reason for installing Python on the Cray is that I
>> want to check out SCons, but it *must* work on the Cray to be of
>> any use to me. At the moment SCons won't work because it requires
>> md5, which couldn't be compiled because it requires a UINT4 data
>> type. So I guess I'll either have to get md5 working or persuade
>> SCons to work without md5 checksums. Oh well...
> 
> Yeah, the md5 code that comes with Python is fairly 32 bit specific.
> However, if you can find some md5-for-cray code in the wild, it's
> probably not too much effort to hook it up to Python.

I'll look into that. Alternatively it should be possible to persuade
SCons to use its TimeStamp module instead of its MD5 module for
file-change detection. But I haven't quite managed to get that going.

Still, it's all great fun!
---
Mark Hadfield            "Ka puwaha et tai nei, Hoea tatou"
m.hadfield at niwa.co.nz
National Institute for Water and Atmospheric Research (NIWA)