[Cython] Shared Cython runtime

[Robert Bradshaw]

On Tue, Apr 9, 2013 at 11:35 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 09.04.2013 20:16:
>> On Tue, Apr 9, 2013 at 8:21 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Stefan Behnel, 09.04.2013 17:12:
>>>> mark florisson, 09.04.2013 16:51:
>>>>> On 9 April 2013 15:47, Stefan Behnel wrote:
>>>>>
>>>>>> mark florisson, 09.04.2013 16:32:
>>>>>>> On 9 April 2013 14:55, Nikita Nemkin wrote:
>>>>>>>> One alternative for code reuse in large Cython projects
>>>>>>>> could be packaging multiple modules into one shared library.
>>>>>>>
>>>>>>> We have 'include'! :) Seriously though, that wouldn't work well with the
>>>>>>> import mechanism, and probably not for C compile time either.
>>>>>>
>>>>>> CPython has support for it. It's actually quite easy, you basically just
>>>>>> register all module init functions explicitly at init time and let CPython
>>>>>> do the rest.
>>>>>>
>>>>>> So, compiling a package into one big Cython module with shared internal
>>>>>> types and utility code should be quite simple and would IMHO be a very cool
>>>>>> feature.
>>>>>
>>>>> I was unaware of that, it'd be a nice feature then (though still not imho
>>>>> the right solution to a runtime system). How does this work? Feel free to
>>>>> point me at some documentation.
>>>>
>>>> It's the inittab mechanism, as described starting here:
>>>>
>>>> http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab
>>>
>>> Oh, and even simpler, you can just stick multiple Cython modules with their
>>> module init functions into one big shared library and have the "main" init
>>> function of that module call the others. That will stick them into
>>> sys.modules as a side effect, so it's not really different from going
>>> through Python's import machinery for each module separately (just way more
>>> efficient - at least if you really need all of the modules...).
>>
>> Yep. We could even create stub .so files for the included ones that
>> did nothing but import the big one to avoid having to worry about
>> importing things in the right order.
>>
>> The big question is what API to specify that things should be linked
>> together. An option to cythonize(...) could be used, but for some
>> projects (e.g. Sage) one would want more control over which modules
>> get bundled together rather than all-or-nothing.
>
> No-one forces you to use a plain cythonize("**/*.pyx"). Just be more
> specific about what files to include in each pattern that you pass to in.
> And you can always call cythonize() more than once if you need to. Once for
> each meta-module should usually be acceptable, given that the bundled
> source modules would tend to have a common configuration anyway.

That would still be painful. Ideally, users should never have to
modify the setup.py file.

> Also, I'd really encourage users to merge things at a package level, rather
> than compiling half of a package into one module and then having separate
> Python or Cython modules lying around next to it. It keeps projects easier
> to understand when you see their source directory layout. Shouldn't be the
> only option, though.

Yes. Actually, this could be a reasonable option for
cythonize("**/*.pyx"), but still wouldn't make sense for Sage (e.g we
have sage/rings and while it would make sense to bundle the common
rings (integers, rationals, reals, etc.) there are others (e.g. the
whole suite of p-adics) that shouldn't be imported by default. Maybe
every exception should really be packages, but that's hard to
change...)

>>> There may be glitches with stuff like "__file__" and friends, but that
>>> should not be any worse than the current way of workings.
>>
>> This can probably be manipulated by Cython, though it's unclear what
>> the best value would be.
>
> I added an option to pyximport that simply uses the file path of the source
> file. It makes sense there, but less so in other contexts. I actually think
> that it's impossible to generally fix in the current infrastructure.
>
> http://bugs.python.org/issue13429
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel