My apologies if this is a duplicate - my first attempt doesn't seem to have
gone back to the list.
---------- Forwarded message ----------
From: Mike Ressler
Mike Ressler wrote:
I'm trying to work with memmaps on very large files, i.e. > 2 GB, up to 10 GB.
Can't believe I'm really the first, but so be it. I just discovered the problem. All the places where
PyObject_As
Buffer is used needs to have the final argument changed to Py_ssize_t (which in arrayobject.h is defined as int if you are using less than Python 2.5). This should be fixed in SVN shortly....
Yeess! My little script can handle everything I've thrown at it now. It can read a 10 GB raw file, strip the top 16 bits, rearrange pixels, byte swap, and write it all back to a 5 GB file in 16 minutes flat. Not bad at all. And I've verified that the output is correct ... If someone can explain the rules of engagement for Lightning Talks, I'm thinking about presenting this at SciPy 2006. Then you'll see there is a reason for my madness. As an aside, the developer pages could use some polish on explaining the different svn areas, and how to get what one wants. An svn checkout as described on the page gets you the 1.1 branch that DOES NOT have the updated memmap fix. After a minute or two of exploring, I found that "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy numpy" got me what I wanted. Thanks for your help and the quick solution. FWIW, I got my copy of the book a couple of weeks ago; very nice. Mike -- mike.ressler@alum.mit.edu -- mike.ressler@alum.mit.edu
Mike Ressler wrote:
My apologies if this is a duplicate - my first attempt doesn't seem to have gone back to the list.
SF if being nasty with GMail. I'll have to speed up moving the list to scipy.org.
If someone can explain the rules of engagement for Lightning Talks, I'm thinking about presenting this at SciPy 2006. Then you'll see there is a reason for my madness.
Unfortunately, we only have scheduled 30 minutes of lightning talks this year. We have twice as many full talks as we did last year. We'll probably only get about 5 or 6 lightning talks clocking in at 5 minutes, tops. In the opening remarks on the first day, we'll tell people to come talk to us (and by "us," I mean "Travis Vaught") during the break and tell us that they want to do a lightning talk about "Foo."
As an aside, the developer pages could use some polish on explaining the different svn areas, and how to get what one wants. An svn checkout as described on the page gets you the 1.1 branch that DOES NOT have the updated memmap fix. After a minute or two of exploring, I found that "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy numpy" got me what I wanted.
Grr. That means developers are not merging changes appropriately. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
participants (2)
-
Mike Ressler -
Robert Kern