[Cython] Sources list handling in Build/Dependencies.py:create_extension_list

[Robert Bradshaw]

On Wed, Mar 9, 2011 at 3:13 AM, Oleksandr Kreshchenko
<cross at ueh0.bank.gov.ua> wrote:
> Hello!
>
> Following the "CEP 201 - Distutils Preprocessing"
> (http://wiki.cython.org/enhancements/distutils_preprocessing)
> there are two possibilities to build a module with cythonize function in
> setup.py.
>
> Second one uses list of distutils.extension.Extension class which accept all
> necessary compiler and
> linker options along with list of non-pyx sources to compile and link with
> for a particular extension module.
> As this way appears more natural for me I do::
>
>    ext_modules = cythonize([Extension(
>        'CyCont', ['CyCont.pyx', 'Cont.c', 'ContCet.c', 'Clash.c',
> 'ContInit.c', 'MemMan.c', '../Log/Log.c'],
>        define_macros = [('CONT_API', '')], # None value expands to 1,
> whereas '' to empty
>        include_dirs = dirs, #libraries = ['libgcov'],
>        extra_compile_args = sys_comp_opts[sys.platform],
>        extra_link_args = sys_link_opts[sys.platform])],
>      show_version = 1, language_level = 3, verbose = 1, include_path = dirs)
>
> But, only CyCont.c is created from CyCont.pyx then compiled and linked into
> the module CyCont.so,
> but the rest of sources to be wrapped are completely ignored in the build
> process.
> So, doing import CyCont I've got undefined symbol error.
>
> However when I move 'CyCont.pyx' to the end of the sources list i.e.::
>
>    ext_modules = cythonize([Extension(
>        'CyCont', ['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c',
> 'MemMan.c', '../Log/Log.c', 'CyCont.pyx'],
>
>    ...
>
> the module is built properly.
>
> Execution flow goes though Cython/Build/Dependencies.py (latest revision)::
>
> 393 def create_extension_list(patterns, exclude=[], ctx=None, aliases=None)
> ...
> 404     for pattern in patterns:
> 405         if isinstance(pattern, str):
> 406             filepattern = pattern
> 407             template = None
> 408             name = '*'
> 409             base = None
> 410             exn_type = Extension
> 411         elif isinstance(pattern, Extension):
> 412             filepattern = pattern.sources[0]
> 413             if os.path.splitext(filepattern)[1] not in ('.py', '.pyx'):
> 414                 # ignore non-cython modules
> 415                 module_list.append(pattern)
> 416                 continue
> 417             template = pattern
> 418             name = template.name
> 419             base = DistutilsInfo(exn=template)
> 420             exn_type = template.__class__
> ...
> 423         for file in glob(filepattern):
> ...
> 437                 module_list.append(exn_type(
> 438                         name=module_name,
> 439                         sources=[file],
> 440                         **kwds))
> ...
> 443     return module_list
>
> If the first list member (sources[0]) is *.pyx or *.py then the only list
> member is CyCont.pyx (see line 439).
>
> Otherwise if CyCont.pyx is placed at the end of the list or, generally, the
> first list member is
> neither *.pyx nor *.py than the loop over patterns/extensions (on line 404)
> breaks at the line 416
> so sources list is left untouched and cythonize function (line 446) does the
> job.
>
> This difference is a bug. Isn't it?

Yes, for sure, thanks for the report.
https://github.com/cython/cython/commit/0fefeb25858ac943a39c4f9ca7e0a493052517a1

> Have "sources[0]" on line 412 and "sources=[file]" on line 439 to be
> revised?
>
>
> BTW, I catch the idea of twiking an extension in setup.py like this::
>
> for ext in ext_modules:
>    ext.define_macros = [('CONT_API', '')]
>    ext.extra_compile_args = sys_comp_opts[sys.platform]
>    ext.extra_link_args = sys_link_opts[sys.platform]
>    ext.sources.extend(['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c',
> 'MemMan.c', '../Log/Log.c'])
>
> but it seams **better** to handle this stuff in the first place within
> Extension(...),
> not to mention using module-level distutils directives.

Yep. The build code is set to handle subclasses of Extension, but
there's only so much you can do, otherwise it becomes difficult to use
two projects that need different distutils hacks.

- Robert