[Pythonmac-SIG] seg fault in intelOSX+python2.4

[David Worrall]

Thanks for your feedback Ronald.
The file is quite large: 4-5 GB
pytables buffers the data and controlling the buffer size  
occasionally stops or slows the rot.
The code has been tested under Linux without incident.
faltet at carabos.com, one of the developers of pytables wrote:

> A Tuesday 21 August 2007, David Worrall escrigué:
>
>>   I lowered NODE_MAX_SLOTS from  256 to128 and that slowed the leak -
>> enough to get some sort of DB happening.
>> It eventually seg faulted, however.
>> I've noticed that sometimes the seg fault causes the (non-python)
>> heap to become corrupted, requiring a HW reboot before pytables
>> becomes useful again.
>> Who knows what memory it may be writing over!
>>
>
> I don't think there is a leak in PyTables itself; it is just that it
> takes a lot of memory to work with many nodes simultaneously.  Perhaps
> reducing the NODE_MAX_SLOTS (to 64, for example) could help a bit  
> more.
> Perhaps it would be a good idea to reduce the memory requirements of
> PyTables nodes (for example, avoid the creation of buffers when they
> are not needed), but this should take quite a few of thought.
>
> Indeed it seems that there is a problem with the MacOSX platform, but
> the fact that this is not reproducible on Linux is unfortunate.
...
> I'm thinking that perhaps you can try with newer versions of Python
> (2.5.1) and HDF5 (the 1.8.0 beta3 is out, and I know that it wears a
> completely revamped cache system, so maybe it is worth a try).
>

Anyway I'll try with gdb as you suggest and report back. I need to  
put it on my task stack so it'll take some time.

thanks again,

David
On 30/08/2007, at 3:48 PM, Ronald Oussoren wrote:

>
> On 30 Aug, 2007, at 1:56, David Worrall wrote:
>
>> Hi all,
>>
>> I've been using PyTables (www.pytables.org) with python2.4 on intel
>> Mac OSX 10.4.10
>> and I'm running into a seg. fault when generating a large hdf5 file.
>>
>> Almost certainly something to do with relationship between OS and
>> python.
>> Has anyone had a similar (memory leak?) experience?
>
> How large a file? It might be a memory management bug in pytables  
> as well.
>
> If you have the developer tools (Xcode & friends) installed you can  
> find out where the crash occurs using gdb.
>
> That is, 'gdb python', then on the prompt for gdb: 'r  
> myscript.py' (adding arguments if needed). When the crash occurs  
> you can use 'where 20' to see the topmost 20 frames on the C stack,  
> or just 'where' to see the entire stack.
>
> Ronald
>>
>>
>>
>>
>> _________________________________________________
>> experimental polymedia:	www.avatar.com.au
>> Sonic Communications Research Group,
>> University of Canberra:	 www.canberra.edu.au/vc-forum/scrg
>> vip=Verbal Interactivity Project
>>
>>
>>
>> _______________________________________________
>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>

_________________________________________________
experimental polymedia:	www.avatar.com.au
Sonic Communications Research Group,
University of Canberra:	 www.canberra.edu.au/vc-forum/scrg
vip=Verbal Interactivity Project