[Patches] [ python-Patches-578297 ] fix for problems with test_longexp

[noreply@sourceforge.net]

Patches item #578297, was opened at 2002-07-07 08:21
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=578297&group_id=5470

Category: Parser/Compiler
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Andrew I MacIntyre (aimacintyre)
>Assigned to: Andrew I MacIntyre (aimacintyre)
Summary: fix for problems with test_longexp 

Initial Comment:
The OS/2 EMX port has long had problems with
test_longexp, which triggers gross memory consumption
on this platform as a result of platform malloc behaviour.

More recently, this appears to have been identified in
MacPython under certain circumstances, although the
problem is apparently more a speed issue than a memory
consumption issue.

The core of the problem is the blizzard of small
mallocs as the parser builds the parse tree and creates
tokens.

The attached patch takes advantage of PyMalloc (built
in by default for 2.3) to insulate the parser from
adverse behaviour in the platform malloc.

The patch has been tested on OS/2 and FreeBSD:
- on OS/2, the patch allows even a system with modest
resources to complete test_longexp successfully and
without swapping to death; on better resourced
machines, the whole regression test is negligibly
slower (0-1%) to complete.  [gcc-2.8.1 -O2]
- on FreeBSD (4.4 tested), test_longexp gains nearly
10%, and completes the whole regression test with a
gain of about 2% (test_longexp is good for about 25% of
the improvement).  [gcc-2.95.3 -O3]
Both platforms are neutral, performance wise, running
MAL's PyBench 1.0.

The patch in its current form is for experimental
evaluation, and not intended for integration into the core.

If there is interest in seeing this integrated, I'd
like feedback on a more elegant way to implement the
functional change.

I've assigned this to Jack for review in the context of
its performance on the Mac.

----------------------------------------------------------------------

>Comment By: Jack Jansen (jackjansen)
Date: 2002-07-08 12:09

Message:
Logged In: YES 
user_id=45365

With Tim's mods test_import and test_longexp now work fine in MacPython. This is both with and without Andrew's patch.

Andrew, I'm assigning back to you, there's little more I can do with this patch. And you'll have to check if you still need it, or whether Tims change to node.c is goo enough for OS/2 as well.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-07-08 08:38

Message:
Logged In: YES 
user_id=31435

Jack, please do a cvs update and try this again.  I checked 
in changes to PyNode_AddChild() that I expect will cure 
your particular woes here.

Andrew, PyMalloc was designed for oodles of small 
allocations.  Feel encouraged to write a patch to change the 
compiler to use PyObject_{Malloc, Realloc, Free} instead.  
Then it will automatically exploit PyMalloc when the latter is 
enabled.

Note that the regression test suite incorporates random 
numbers in several tests, and in ways that can affect 
runtime.  Small differences in aggregate test suite runtime 
are meaningless because of this.

----------------------------------------------------------------------

Comment By: Jack Jansen (jackjansen)
Date: 2002-07-07 23:24

Message:
Logged In: YES 
user_id=45365

Unfortunately on the Mac it doesn't help anything for the test_longexp problem, nor for the similar test_import problem.

The problem with MacPython's malloc seems to be that large reallocs cause the slowdown. And the addchild() calls will continually realloc a block of memory to a slightly larger size (I gave up when it was about 800KB, after a minute or two, and growing at tens of KB per second). As soon as the block is larger than SMALL_REQUEST_TRESHOLD pymalloc will simply call the underlying system malloc/realloc.

----------------------------------------------------------------------

Comment By: Andrew I MacIntyre (aimacintyre)
Date: 2002-07-07 08:41

Message:
Logged In: YES 
user_id=250749

Oops.  On FreeBSD,  test_longexp contributes 15% of the
performance gain (not 25%) observed for the regression test
with the patch applied.

Also, I would expect to make this a platform specific change
if its integrated, rather than a general change (unless that
it is seen as more appropriate).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=578297&group_id=5470