Hi! Do numpy memmap have a way of explicitly flushing data to disk and/or closing the memmap. In numarray these were methods called memmappedArr.flush() and memmappedArr.close() Thanks, Sebastian
Sebastian Haase wrote:
Hi! Do numpy memmap have a way of explicitly flushing data to disk and/or closing the memmap.
There is a sync method that performs the flush. To close the memmap, delete it. More detail: The memmap sub-class has a _mmap attribute that is the Python memory-map object. -Travis
On 1/31/07, Travis Oliphant <oliphant.travis@ieee.org> wrote:
Sebastian Haase wrote:
Hi! Do numpy memmap have a way of explicitly flushing data to disk and/or closing the memmap.
There is a sync method that performs the flush. To close the memmap, delete it.
More detail: The memmap sub-class has a _mmap attribute that is the Python memory-map object.
Quick question out of ignorance: shouldn't the API offer an explicit close method? The reason is that in python, doing 'del foo' doesn't ensure real deletion of the underlying object that will fire the close methods, since you may easily have other names pointing to the same object. Having an explicit, guaranteed way to call .close() sounds like a good thing to me, but I may well be missing something. Cheers, f
Fernando Perez wrote:
On 1/31/07, Travis Oliphant <oliphant.travis@ieee.org> wrote:
Sebastian Haase wrote:
Hi! Do numpy memmap have a way of explicitly flushing data to disk and/or closing the memmap.
There is a sync method that performs the flush. To close the memmap, delete it.
More detail: The memmap sub-class has a _mmap attribute that is the Python memory-map object.
Quick question out of ignorance: shouldn't the API offer an explicit close method? The reason is that in python, doing 'del foo' doesn't ensure real deletion of the underlying object that will fire the close methods, since you may easily have other names pointing to the same object. Having an explicit, guaranteed way to call .close() sounds like a good thing to me, but I may well be missing something.
I don't know. If you have other things pointing to it, should you really close it? At any rate you have access to the mmap file through the _mmap attribute. So, you can always do self._mmap.close() -Travis
On 1/31/07, Travis Oliphant <oliphant@ee.byu.edu> wrote:
Fernando Perez wrote:
I don't know. If you have other things pointing to it, should you really close it?
Well, it's like a file: you can close it because you've decided it's time to close it, and I think it's better that other references get an exception if they try to write to it when they shouldn't: In [2]: file1 = open('foo','w') In [3]: file1.write('Some text') In [4]: file2 = file1 In [5]: file1.close() In [6]: file2.write("I'm not the original owner but I'll try to write anyway") --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/fperez/<ipython console> ValueError: I/O operation on closed file This seems like the right API to me.
At any rate you have access to the mmap file through the _mmap attribute. So, you can always do
self._mmap.close()
I don't like an API that encourages access to internal semi-private members, and it seems to me that closing the object is a reasonably top-level operation whose impmlementation details (in this case the existence and name of the _mmap member) should be encapsulated out. Cheers, f
On 1/31/07, Fernando Perez <fperez.net@gmail.com> wrote:
On 1/31/07, Travis Oliphant <oliphant@ee.byu.edu> wrote:
Fernando Perez wrote:
I don't know. If you have other things pointing to it, should you really close it?
Well, it's like a file: you can close it because you've decided it's time to close it, and I think it's better that other references get an exception if they try to write to it when they shouldn't:
In [2]: file1 = open('foo','w')
In [3]: file1.write('Some text')
In [4]: file2 = file1
In [5]: file1.close()
In [6]: file2.write("I'm not the original owner but I'll try to write anyway") --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last)
/home/fperez/<ipython console>
ValueError: I/O operation on closed file
This seems like the right API to me.
At any rate you have access to the mmap file through the _mmap attribute. So, you can always do
self._mmap.close()
I don't like an API that encourages access to internal semi-private members, and it seems to me that closing the object is a reasonably top-level operation whose impmlementation details (in this case the existence and name of the _mmap member) should be encapsulated out.
Cheers,
f
After asking this question rather for acedemical reasons, I now realize that I indead must have a "somehow dangling" (i.e. not deleted) reference problem. I create my Medical-Image-File-class object on a 400MB file repeatedly (throwing the result away) and after 5 calles I get: File "/jws30/haase/PrLinN/Priithon/Mrc.py", line 55, in __init__ self.m = N.memmap(path, mode=mode) File "/home/haase/qqq/lib/python/numpy/core/memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) EnvironmentError: [Errno 12] Cannot allocate memory Calling gc.collect() seams to clean things up and I can create 4-5 times afterwards, before running out of memory space again. Note: My code is based on code that was tested and worked using numarray. Thanks, Sebastian
On 1/31/07, Fernando Perez <fperez.net@gmail.com> wrote:
On 1/31/07, Travis Oliphant <oliphant@ee.byu.edu> wrote:
Fernando Perez wrote:
I don't know. If you have other things pointing to it, should you really close it?
Well, it's like a file: you can close it because you've decided it's time to close it, and I think it's better that other references get an exception if they try to write to it when they shouldn't:
In [2]: file1 = open('foo','w')
In [3]: file1.write('Some text')
In [4]: file2 = file1
In [5]: file1.close()
In [6]: file2.write("I'm not the original owner but I'll try to write anyway") --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last)
/home/fperez/<ipython console>
ValueError: I/O operation on closed file
This seems like the right API to me.
At any rate you have access to the mmap file through the _mmap attribute. So, you can always do
self._mmap.close()
I don't like an API that encourages access to internal semi-private members, and it seems to me that closing the object is a reasonably top-level operation whose impmlementation details (in this case the existence and name of the _mmap member) should be encapsulated out.
After asking this question rather for acedemical reasons, I now realize that I indead must have a "somehow dangling" (i.e. not deleted) reference problem. I create my Medical-Image-File-class object on a 400MB file repeatedly (throwing the result away) and after 5 calles I get: File "/jws30/haase/PrLinN/Priithon/Mrc.py", line 55, in __init__ self.m = N.memmap(path, mode=mode) File "/home/haase/qqq/lib/python/numpy/core/memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) EnvironmentError: [Errno 12] Cannot allocate memory Calling gc.collect() seams to clean things up and I can create 4-5 times afterwards, before running out of memory space again. Note: My code is based on code that was tested and worked using numarray. Thanks, Sebastian
participants (5)
-
Fernando Perez -
Sebastian Haase -
Sebastian Haase -
Travis Oliphant -
Travis Oliphant