[Cython] compiler performance issue for extended utility code

[mark florisson]

On 8 October 2011 13:10, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/10/8 mark florisson <markflorisson88 at gmail.com>:
>> On 8 October 2011 08:03, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Vitja Makarov, 07.10.2011 18:01:
>>>>>
>>>>> 2011/10/7 Stefan Behnel:
>>>>>>
>>>>>> Vitja Makarov, 06.10.2011 23:12:
>>>>>>>
>>>>>>> Here is small comparison on compiling urllib.py with cython:
>>>>>>>
>>>>>>> ((e8527c5...)) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
>>>>>>> ../cython.py urllib.py
>>>>>>>
>>>>>>> real    0m1.699s
>>>>>>> user    0m1.650s
>>>>>>> sys     0m0.040s
>>>>>>> (master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
>>>>>>> ../cython.py urllib.py
>>>>>>>
>>>>>>> real    0m2.830s
>>>>>>> user    0m2.790s
>>>>>>> sys     0m0.030s
>>>>>>>
>>>>>>>
>>>>>>> It's about 1.5 times slower.
>>>>>>
>>>>>> That's a pretty serious regression for
>>>>>> plain Python code then. Again, this needs proper profiling.
>>>>
>>>> I've added return statement on top of CythonScope.test_cythonscope,
>>>> now I have these timings:
>>>>
>>>> (master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
>>>> ../cython.py urllib.py
>>>>
>>>> real    0m1.764s
>>>> user    0m1.700s
>>>> sys     0m0.060s
>>>
>>> Ok, then it's only a bug. "create_testscope" is on by default in Main.py,
>>> Context.__init__(). I don't know what it does exactly, but my guess is that
>>> the option should a) be off by default and b) should rather be passed in by
>>> the test runner as part of the compile options rather than being a parameter
>>> of the Context class. AFAICT, it's currently only used in TreeFragment.py,
>>> where it is being switched off explicitly for parsing code snippets.
>>>
>>> Stefan
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> It turns it off to avoid infinite recursion. This basically means that
>> you cannot use stuf from the Cython scope in your Cython utilities. So
>> in your Cython utilities, you have to declare the C version of it
>> (which you declared with the @cname decorator).
>>
>> This is not really something that can just be avoided loading like
>> this. Perhaps one solution could be to load the test scope when you do
>> a lookup in the cython scope for which no entry is found. But really,
>> libcython and serializing entries will solve all this, so I suppose
>> the real question is, do we want to do a release before we support
>> such functionality?
>> Anyway, the cython scope lookup would be a simple hack worth a try.
>>
>
> Does utility code supports something like dependencies? And could that
> help here?

Yeah they can have dependencies like normal UtilitieCodes.

> I've also noticed that some utilities are loaded unconditionally
> perhaps it's better to introduce lazy loading.

Well, they shouldn't be. If they are it's generally a bug. I noticed
that it happens in the test runner though, although it should create a
fresh context with freshly initialized entries.

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>