Building Python on Cray T3E

[Mark Hadfield]

"Michael Hudson" <mwh at python.net> wrote in message
news:lkhelg74ts.fsf at pc150.maths.bris.ac.uk...

> So does the resulting Python actually work after this?  We got this
> far with 2.2.1c1, but you never told me if the dang thing worked or
> not...

I'm not sure who this question was directed to, but here are the
results of my attempts to build Python 2.2.1 on a Cray T3E over the
last few days. The summary is: it can be built and installed and it
runs several simple scripts, but several modules are broken to a
greater or lesser extent. It's not yet clear to me how useful it will
be or how much work is required to fix things.

The procedure is the usual "configure; make; make install" with the
following variations and notes:

  - The compiler (found automatically by the configure script) is
  Cray's cc. We don't have gcc on the NIWA machine and a note from
  Konrad Hinsen in README warns against using it anyway.

  - As has been discussed before on this group, Python's unicode
  module cannot be built because the Cray compiler lacks a 16-bit
  character datatype (something like that--don't trust me on the
  details here). So we disable it with "configure
  --enable-unicode=ucs4".

  - As noted earlier in this thread, there is a C-function name clash
  in the _sre module, which can be easily fixed. However, recall that
  sre is one of two alternative regular-expression engines for the re
  module, the other engine being pre. It turns out that sre does not
  pass its tests so the path of least resistance is is to modify
  Lib/re.py to use the pre engine (the change is trivial) and comment
  out the _sre entry in Modules/Setup.

  - The Cray compiler does not support dynamic linking, so extensions
  must be explicitly listed in Modules/Setup. Here is the list of
  extensions I have built & linked successfully:

    posix posixmodule.c
    new newmodule.c
    array arraymodule.c
    cmath cmathmodule.c
    math mathmodule.c
    struct structmodule.c
    time timemodule.c
    operator operator.c
    _weakref _weakref.c
    _codecs _codecsmodule.c
    _testcapi _testcapimodule.c
    _symtable symtablemodule.c
    strop stropmodule.c
    regex regexmodule.c regexpr.c
    pcre pcremodule.c pypcre.c
    _locale _localemodule.c
    fcntl fcntlmodule.c
    pwd pwdmodule.c
    grp grpmodule.c
    errno errnomodule.c
    select selectmodule.c
    xreadlines xreadlinesmodule.c
    _socket socketmodule.c
    crypt cryptmodule.c
    termios termios.c
    timing timingmodule.c
    rotor rotormodule.c
    syslog syslogmodule.c
    dbm dbmmodule.c
    binascii binascii.c
    parser parsermodule.c
    cStringIO cStringIO.c
    cPickle cPickle.c

  - With the above modules included, Python is built
  successfully. There are quite a few warnings, which I haven't
  checked out yet. I think that some of them come home to roost in the
  tests.

  - After building Python, make runs setup.py, which attempts (among
  other things) to build extensions not already built & linked
  statically. This is a futile but harmless exercise; the inevitable
  failures do not cause make to abort. I have seen elsewhere (in
  connection with Cygwin Python) a reference to a "--disable-shared"
  option to configure that I presume would inhibit this. However this
  is ignored by configure in Python 2.2.1.

  - Testing is a somewhat depressing experience. On the NIWA machine,
  a simple "make test" command fails to complete after 12 hours and
  thrashes the swap space, with the side effect of inhibiting all
  network access for 10 seconds out of every 30. (They don't call
  these things supercomputers for nothing. I bet you can't do *that*
  on your $1000 Linux box!) By running "make test" for a while then
  completing the tests one at a time, I have established that the
  following tests fail:

    test test_al skipped -- No module named al
    test test_audioop skipped -- No module named audioop
    test test_binhex failed -- CRC error
    test test_bsddb skipped -- No module named bsddb
    test test_cd skipped -- No module named cd
    test test_cl skipped -- No module named cl
    test test_curses skipped -- No module named _curses
    test test_descr skipped -- No module named xxsubtype
    test test_dl skipped -- No module named dl
    test test_fcntl crashed -- IO Error
    test test_fpformat failed -- AssertionError: '-0' != '0'
    test test_hmac failed -- No module named md5
    test test_hotshot skipped -- No module named _hotshot
    test test_imageop skipped -- No module named imageop
    test test_imgfile skipped -- No module named imgfile
    test test_linuxaudiodev skipped -- No module named linuxaudiodev
    test test_locale skipped -- test locale en_US not supported
    test test_longexp crashed -- exceptions.MemoryError:
    test test_md5 skipped -- No module named md5
    test test_minidom skipped -- No module named pyexpat
    test test_mmap skipped -- No module named mmap
    test test_nis skipped -- No module named nis
    test test_ntpath skipped -- No module named nt
    test test_openpty skipped -- No openpty() available.
    test test_pep247 skipped -- No module named md5
    test test_poll skipped -- select.poll not defined
    test test_pty skipped -- Pseudo-terminals (seemingly) not
      functional.
    test test_pyexpat skipped -- No module named pyexpat
    test test_re crashed -- exceptions.TypeError: 'NoneType' object is
      not callable
    test test_rgbimg skipped -- No module named rgbimg
    test test_rotor produced unexpected output:
    test test_sax skipped -- no XML parsers available
    test test_sha skipped -- No module named sha
    test test_socket crashed -- socket.error: unknown address family
    test test_socket_ssl skipped -- Use of the `network' resource
      not enabled
    test test_socketserver skipped -- Use of the `network' resource
      not enabled
    test test_sre crashed -- exceptions.UnicodeError: \N escapes not
      supported (can't load unicodedata module)
    test test_sunaudiodev skipped -- No module named sunaudiodev
    test test_sundry skipped -- No module named md5
    test test_ucn skipped -- No module named unicodedata
    test test_unicode crashed -- exceptions.UnicodeError: \N escapes
      not supported (can't load unicodedata module)
    test test_unicode_file skipped -- No Unicode filesystem semantics
      on this platform.
    test test_unicodedata skipped -- No module named sha
    test test_urllib2 skipped -- No module named md5
    test test_winreg skipped -- No module named winreg
    test test_winsound skipped -- No module named winsound
    test test_zipfile skipped -- No module named zlib
    test test_zlib skipped -- No module named zlib

  Not all of the test failures are of any significance, of course. (I
  didn't really expect Windows registry operations to work.) But the
  failure in test_re is a little worrying...

My immediate reason for installing Python on the Cray is that I want
to check out SCons, but it *must* work on the Cray to be of any use to
me. At the moment SCons won't work because it requires md5, which
couldn't be compiled because it requires a UINT4 data type. So I guess
I'll either have to get md5 working or persuade SCons to work without
md5 checksums. Oh well...

--
Mark Hadfield            "Ka puwaha et tai nei, Hoea tatou"
m.hadfield at niwa.co.nz
National Institute for Water and Atmospheric Research (NIWA)