[Cython] compiler performance issue for extended utility code

[mark florisson]

On 8 October 2011 10:22, mark florisson <markflorisson88 at gmail.com> wrote:

> On 8 October 2011 08:03, Stefan Behnel <stefan_ml at behnel.de> wrote:
> > Vitja Makarov, 07.10.2011 18:01:
> >>>
> >>> 2011/10/7 Stefan Behnel:
> >>>>
> >>>> Vitja Makarov, 06.10.2011 23:12:
> >>>>>
> >>>>> Here is small comparison on compiling urllib.py with cython:
> >>>>>
> >>>>> ((e8527c5...)) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
> >>>>> ../cython.py urllib.py
> >>>>>
> >>>>> real    0m1.699s
> >>>>> user    0m1.650s
> >>>>> sys     0m0.040s
> >>>>> (master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
> >>>>> ../cython.py urllib.py
> >>>>>
> >>>>> real    0m2.830s
> >>>>> user    0m2.790s
> >>>>> sys     0m0.030s
> >>>>>
> >>>>>
> >>>>> It's about 1.5 times slower.
> >>>>
> >>>> That's a pretty serious regression for
> >>>> plain Python code then. Again, this needs proper profiling.
> >>
> >> I've added return statement on top of CythonScope.test_cythonscope,
> >> now I have these timings:
> >>
> >> (master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
> >> ../cython.py urllib.py
> >>
> >> real    0m1.764s
> >> user    0m1.700s
> >> sys     0m0.060s
> >
> > Ok, then it's only a bug. "create_testscope" is on by default in Main.py,
> > Context.__init__(). I don't know what it does exactly, but my guess is
> that
> > the option should a) be off by default and b) should rather be passed in
> by
> > the test runner as part of the compile options rather than being a
> parameter
> > of the Context class. AFAICT, it's currently only used in
> TreeFragment.py,
> > where it is being switched off explicitly for parsing code snippets.
> >
> > Stefan
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > http://mail.python.org/mailman/listinfo/cython-devel
> >
>
> It turns it off to avoid infinite recursion. This basically means that
> you cannot use stuf from the Cython scope in your Cython utilities. So
> in your Cython utilities, you have to declare the C version of it
> (which you declared with the @cname decorator).
>
> This is not really something that can just be avoided loading like
> this. Perhaps one solution could be to load the test scope when you do
> a lookup in the cython scope for which no entry is found. But really,
> libcython and serializing entries will solve all this, so I suppose
> the real question is, do we want to do a release before we support
> such functionality?
> Anyway, the cython scope lookup would be a simple hack worth a try.
>

I applied the hack, i.e. defer loading the scope until the first entry in
the cython scope can't be found:
https://github.com/markflorisson88/cython/commit/ad4cf6303d1bf8a81e3afccc9572559a34827a3b

[0] [11:16] ~  ➤ time cython urllib.py # conditionally load scope
cython urllib.py  2.75s user 0.14s system 99% cpu 2.893 total
[0] [11:17] ~  ➤ time cython urllib.py # always load scope
cython urllib.py  4.08s user 0.16s system 99% cpu 4.239 total
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20111009/0436bf6f/attachment.html>