[Cython] dynamically compile and import pure python modules

[Stefan Behnel]

mark florisson, 28.04.2012 20:18:
> On 28 April 2012 18:55, Stefan Behnel wrote:
>> mark florisson, 27.04.2012 22:38:
>>> On 27 April 2012 21:16, Stefan Behnel wrote:
>>>> What about this deal: we remove the hard bootstrap dependency on the fused
>>>> types code (and maybe other Cython specific features) and require its
>>>> compilation at install time in Py2.4 (and maybe even 2.5). That would allow
>>>> us to use newer Python syntax (and even Cython supported syntax) there
>>>> (except for fused types, obviously). Failure to compile the module in
>>>> Python 2.4/5 at install time would then abort the installation. Bad luck
>>>> for the user, but easy to fix by installing a newer Python version.
>>>>
>>>> That would give us the advantage of not needing to pollute user home
>>>> directories with shared libraries at runtime (which I would consider a very
>>>> annoying property).
>>>
>>> I think it's fine to require compiling in the installed case (or will
>>> that be a problem for some package managers?). In the non-installed
>>> case with python versions smaller than needed, would you prefer a
>>> pyximport or an error message telling you to install Cython? Because
>>> for development you really don't want to install every time.
>>
>> I think it's fine to require at least Python 2.6 for Cython core
>> development. Just the installation (basically, what we test in Jenkins
>> anyway) should work in Py2.4 and shouldn't require any rebuilds at runtime.
> 
> Well, sometimes stuff works in say, 2.7 but fails in 2.4. In that case
> you really have to test with the failing python versions, which means
> you'd have to reinstall every time you want to try the tests again
> (this is handled automatically for py3k, which runs the 2to3 tool).

The number of times I recently ran tests in Py2.4 myself is really not
worth mentioning. Most of the time, when something fails there, the error I
get in Jenkins is so obvious that I just commit an untested fix for it.

I think it's really acceptable to require a run of "setup.py build_ext -i"
for local developer testing in Py2.4.


> I'm also sure many users just clone from git, add the directory to
> PYTHONPATH and work from there.

I'm sure there are close to no users who try to do that with Py2.4 these
days. Maybe there are some who do it with Py2.5, but we are not currently
considering to break a plain Python run there, AFAICT.

I think the normal way users employ Cython is after a proper installation.


> So I guess what I'm saying is, it's fine to mandate compilation at
> compile time (don't allow flags to disable compilation), and (for me),
> pyximport is totally fine, but Cython must be workable (all
> functionality), without needing to install or build, in all versions.

Workable, ok. But if fused types are only available in a compiled installed
version under Python 2.4, that's maybe not optimal but certainly
acceptable. Users of Py2.4 should be used to suffering anyway.


> That means either not using the with statement, or compiling with
> pyximport in certain versions in certain situations only (i.e., only
> in incompatible python version in case the user neither built nor
> installed Cython). I don't think that's a problem, if people don't
> like to have those shared library modules (will they even notice?) in
> their user directories, they can install or build Cython.

Why require the use of pyximport at runtime when we can do everything
during installation? I really don't see an advantage.


> Finally, my attachment to the with statement here is mostly to the
> aesthetics of the resulting code, rewriting my pull request is not so
> much work, so we can leave that out of consideration here.

If it's not too much work, that would obviously make things go a lot smoother.

Stefan