[Cython] compiler performance issue for extended utility code

[Vitja Makarov]

2011/10/8 mark florisson <markflorisson88 at gmail.com>:
> On 8 October 2011 08:03, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Vitja Makarov, 07.10.2011 18:01:
>>>>
>>>> 2011/10/7 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 06.10.2011 23:12:
>>>>>>
>>>>>> Here is small comparison on compiling urllib.py with cython:
>>>>>>
>>>>>> ((e8527c5...)) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
>>>>>> ../cython.py urllib.py
>>>>>>
>>>>>> real    0m1.699s
>>>>>> user    0m1.650s
>>>>>> sys     0m0.040s
>>>>>> (master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
>>>>>> ../cython.py urllib.py
>>>>>>
>>>>>> real    0m2.830s
>>>>>> user    0m2.790s
>>>>>> sys     0m0.030s
>>>>>>
>>>>>>
>>>>>> It's about 1.5 times slower.
>>>>>
>>>>> That's a pretty serious regression for
>>>>> plain Python code then. Again, this needs proper profiling.
>>>
>>> I've added return statement on top of CythonScope.test_cythonscope,
>>> now I have these timings:
>>>
>>> (master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
>>> ../cython.py urllib.py
>>>
>>> real    0m1.764s
>>> user    0m1.700s
>>> sys     0m0.060s
>>
>> Ok, then it's only a bug. "create_testscope" is on by default in Main.py,
>> Context.__init__(). I don't know what it does exactly, but my guess is that
>> the option should a) be off by default and b) should rather be passed in by
>> the test runner as part of the compile options rather than being a parameter
>> of the Context class. AFAICT, it's currently only used in TreeFragment.py,
>> where it is being switched off explicitly for parsing code snippets.
>>
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> It turns it off to avoid infinite recursion. This basically means that
> you cannot use stuf from the Cython scope in your Cython utilities. So
> in your Cython utilities, you have to declare the C version of it
> (which you declared with the @cname decorator).
>
> This is not really something that can just be avoided loading like
> this. Perhaps one solution could be to load the test scope when you do
> a lookup in the cython scope for which no entry is found. But really,
> libcython and serializing entries will solve all this, so I suppose
> the real question is, do we want to do a release before we support
> such functionality?
> Anyway, the cython scope lookup would be a simple hack worth a try.
>

Does utility code supports something like dependencies? And could that
help here?

I've also noticed that some utilities are loaded unconditionally
perhaps it's better to introduce lazy loading.

-- 
vitja.