[Distutils] Disposition of C extensions and packages

[Greg Ward]

On 18 December 1999, A.M. Kuchling said:
> [Crossposted to xml-sig, distutils-sig]
> 
> I'm working on getting the XML-SIG's CVS tree to install using the
> current version of the Distutils.  Right now there are two C
> extensions, sgmlop.so and pyexpat.so, and they're installed under
> xml/parsers/ .  It's hard to handle this case using the distutils code
> as it stands, because it expects to put extensions into a
> build/platlib/ directory, from which they'll be installed into
> site-packages.  

Well, I missed this thread first time around because I was on holiday.
Andrew went to such amazing lengths to do something that's actually
quite easy that I thought it would be worth following up, even if I'm a
month late.

In short, putting Python extension modules into a package is really,
really, *really* easy.  Here's what *was* in Andrew's setup.py for the
XML package (edited for readability):

  ext_modules = [('sgmlop',
                   { 'sources' : ['extensions/sgmlop.c']
                                        }),
                 ('pyexpat',
                   { 'define': [('XML_NS', None)],
                     'include_dirs': [ 'extensions/expat/xmltok',
                                       'extensions/expat/xmlparse' ], 
                     'sources' : [...]
                   }
                  )
                ]

Note that the names of the two extensions are 'sgmlop' and 'pyexpat'.
In regards to where the .so files wind up, *the name is the most
important thing* (and usually all you need).  All we had to do to get
rid of Andrew's extended command classes (and other kludgery) was set
the names of the two extension modules appropriately, so the XML setup
script now has this:

  ext_modules = [('xml.parsers.sgmlop',
                   { 'sources' : ['extensions/sgmlop.c']
                                        }),
                 ('xml.parsers.pyexpat',
                   { 'define': [('XML_NS', None)],
                     'include_dirs': [ 'extensions/expat/xmltok',
                                       'extensions/expat/xmlparse' ], 
                     'sources' : [...]
                   }
                  )
                ]

i.e. the names of the extensions had to change... and nothing else.
(Well, we commented out the unnecessarily extended command classes.)

I assumed that large module distributions with many extensions will
probably put them in the same package.  Thus, we could have done the
change this way as well:

  ext_package = 'xml.parsers'
  ext_modules = [('sgmlop',
                   { 'sources' : ['extensions/sgmlop.c']
                                        }),
                 ('pyexpat',
                   { 'define': [('XML_NS', None)],
                     'include_dirs': [ 'extensions/expat/xmltok',
                                       'extensions/expat/xmlparse' ], 
                     'sources' : [...]
                   }
                  )
                ]

And if you only have a common base package for many extensions, you
should still be able to specify that base package with 'ext_package':

  ext_package = 'xml.parsers'
  ext_modules = [('pkg1.sgmlop',
                   { 'sources' : ['extensions/sgmlop.c']
                                        }),
                 ('pkg2.pyexpat',
                   { 'define': [('XML_NS', None)],
                     'include_dirs': [ 'extensions/expat/xmltok',
                                       'extensions/expat/xmlparse' ], 
                     'sources' : [...]
                   }
                  )
                ]

If everything goes as planned, that should result in two extensions
called 'xml.parsers.pkg1.sgmlop' and 'xml.parsers.pkg2.pyexpat'.

Note that my assertions about these last two examples are based solely
on reading the code and a dim recollection of what I had in mind when I
wrote it -- i.e. I haven't tested them.  YMMV, but please let me know if
it does.

        Greg