is deinitialisation of embedded python equal to full restart?
Hello. Sorry if posting to wrong maillist, I have read full list and decided that this one is the most appropriate.
I have a quick question: am I able to do multiple initialisations/deinitialisations of embedded python interpreter? I mean - I'd like to init python, run a script in it for some time (say, couple of hours) and then just deinitialise python so everything, including the python program that didn't expected such brutal end to be freed and deinitialised. Then, after some more time, call Py_Initialize() again and again load and start using my python program.
The most essential for me is freeing all sorts of RAM that python allocated for objects or whatever else. IOW - I want python to be returned in virgin state as it was on my program load w/o exiting my program and closing all files/tcp connections that it have opened/or resetting any of other my C data structures that were created.
Is that possible?
-- Respectfully Alexey Nezhdanov
Alexey Nezhdanov snakeru@gmail.com writes:
The most essential for me is freeing all sorts of RAM that python allocated for objects or whatever else.
Reinitializing Python should work, but it might not produce the effect you expect.
Python allocates memory using the underlying C functions such as malloc. Shutting down the interpreter simply makes Python call free on the memory segments that belong to Python objects. Because of the way malloc and free work, doing that is no guarantee that memory will be returned to the operating system. free() only guarantees that the memory will be marked as usable for unspecified later calls to malloc. Some malloc implementations go one step further and actually return the memory to the system, but that is only possible when freeing the block(s) at the very end of the region, so you cannot rely on it.
At the end of the day, your process will probably not decrease in memory footprint; however, it will not grow on further allocations because it will reuse the freed memory. But, barring a bug in Python, that is exactly the situation you would be in if you didn't reinitialize the interpreter in the first place.
IOW - I want python to be returned in virgin state as it was on my program load w/o exiting my program and closing all files/tcp connections that it have opened/or resetting any of other my C data structures that were created.
Doesn't Python automatically close open files when the script finishes?
On Friday 13 July 2007 14:29, Hrvoje Niksic wrote:
Alexey Nezhdanov snakeru@gmail.com writes:
The most essential for me is freeing all sorts of RAM that python allocated for objects or whatever else.
Reinitializing Python should work, but it might not produce the effect you expect.
Python allocates memory using the underlying C functions such as malloc. Shutting down the interpreter simply makes Python call free on the memory segments that belong to Python objects. Because of the way malloc and free work, doing that is no guarantee that memory will be returned to the operating system. free() only guarantees that the memory will be marked as usable for unspecified later calls to malloc. Some malloc implementations go one step further and actually return the memory to the system, but that is only possible when freeing the block(s) at the very end of the region, so you cannot rely on it.
At the end of the day, your process will probably not decrease in memory footprint; however, it will not grow on further allocations because it will reuse the freed memory. But, barring a bug in Python, that is exactly the situation you would be in if you didn't reinitialize the interpreter in the first place. ok, thanks. Still it may be ok for my purposes if I can keep python from growing forever and just keep it beneath some limit. Also, as I understand - memory that is marked as available for reuse will be swapped out if need for RAM will arise.
IOW - I want python to be returned in virgin state as it was on my program load w/o exiting my program and closing all files/tcp connections that it have opened/or resetting any of other my C data structures that were created.
Doesn't Python automatically close open files when the script finishes? Yes, it does. But I meant files that are opened by C part of the program.
-- Respectfully Alexey Nezhdanov
Alexey Nezhdanov snakeru@gmail.com writes:
But, barring a bug in Python, that is exactly the situation you would be in if you didn't reinitialize the interpreter in the first place. ok, thanks. Still it may be ok for my purposes if I can keep python from growing forever and just keep it beneath some limit.
But Python should behave like that anyway. If your Python is "growing forever", it means that there is a bug in Python, or in the code using it incorrectly.
Also, as I understand - memory that is marked as available for reuse will be swapped out if need for RAM will arise.
Theoretically. In practice, if the memory is interspersed with the chunks of memory that are in use, you get mixed results.
Hrvoje Niksic wrote:
Alexey Nezhdanov snakeru@gmail.com writes:
But, barring a bug in Python, that is exactly the situation you would be in if you didn't reinitialize the interpreter in the first place. ok, thanks. Still it may be ok for my purposes if I can keep python from growing forever and just keep it beneath some limit.
But Python should behave like that anyway. If your Python is "growing forever", it means that there is a bug in Python, or in the code using it incorrectly.
A different but somewhat related problem with Python is that the underlying C memory manager does not release memory back to the operating system when Python releases it to the memory manager. This leads to a situation where you have a momentary need for more memory, say at startup, and then you drop back down in usage but never give that memory to other apps.
This is a known problem with the use of Python in embedded devices such as the One-Laptop-Per-Child and the OpenMoko cell phone. I've heard rumors that someone is looking at it, especially because of the OLPC project but have no other information.
The Parrot/Pynie project (Python on top of the Perl6 engine) is using memory pools and mark/sweep so they can rearrange objects behind the scenes and release pools back to the OS. Not that that helps us cPython users of course, but I thought it interesting.
-Jeff
On 2007-07-16 00:58, Jeff Rush wrote:
Hrvoje Niksic wrote:
Alexey Nezhdanov snakeru@gmail.com writes:
But, barring a bug in Python, that is exactly the situation you would be in if you didn't reinitialize the interpreter in the first place. ok, thanks. Still it may be ok for my purposes if I can keep python from growing forever and just keep it beneath some limit. But Python should behave like that anyway. If your Python is "growing forever", it means that there is a bug in Python, or in the code using it incorrectly.
A different but somewhat related problem with Python is that the underlying C memory manager does not release memory back to the operating system when Python releases it to the memory manager. This leads to a situation where you have a momentary need for more memory, say at startup, and then you drop back down in usage but never give that memory to other apps.
This is a known problem with the use of Python in embedded devices such as the One-Laptop-Per-Child and the OpenMoko cell phone. I've heard rumors that someone is looking at it, especially because of the OLPC project but have no other information.
The Parrot/Pynie project (Python on top of the Perl6 engine) is using memory pools and mark/sweep so they can rearrange objects behind the scenes and release pools back to the OS. Not that that helps us cPython users of course, but I thought it interesting.
Python 2.5 does the same. Tim Peters (AFAIR) applied some changes to pymalloc to make this possible.
Note that you can tune pymalloc using various parameters you can tweak (see obmalloc.c) and even switch it off if you need the OS to be in complete control. Doing so will slow the interpreter considerably.
Also note that quite a few object implementations use free lists which are not easy to tweak in this way.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, Jul 16 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
participants (4)
-
Alexey Nezhdanov -
Hrvoje Niksic -
Jeff Rush -
M.-A. Lemburg