[Numpy-discussion] Loading a > GB file into array

[David Cournapeau]

Martin Spacek wrote:
> Gael Varoquaux wrote:
>> Very interesting. Have you made measurements to see how many times you
>> lost one of your cycles. I made these kind of measurements on Linux using
>> the real-time clock with C and it was very interesting (
>> http://www.gael-varoquaux.info/computers/real-time ). I want to redo them
>> with Python, as I except to have similar results with Python. It would be
>> interesting to see how Windows fits in the picture (I know nothing about
>> Windows, so I really can't make measurements on Windows).
>
> Neat, thanks for that, I'll have a look. I'm very slowly transitioning 
> my computing "life" over to linux, but I've been told by Andrew Straw (I 
> think http://visionegg.org might have some details, see the mailing 
> list) that it's harder to get close to a real-time OS with linux (while 
> running high level stuff like opengl and python) than it is in windows.
My impression is that is is more like the contrary; linux implements 
many  posix facilities for more 'real-time' behaviour: it implements a 
FIFO scheduler, you have mlock facilities to avoid paging, etc... and of 
course, you can control your environment much more easily (one buggy 
driver can kill the whole thing as far as latency is concerned, for 
example). I did not find those info you are talking about on visionegg  ?

Now, for python, this is a different matter. In you need to do things in 
real-time, setting a high priority is not enough, and python has several 
characteristics which make it less than suitable for real-time (heavy 
usage of memory allocation, garbage collector, etc...). I guess that 
when you do things every few ms, with enough memory (every ms gives you 
millions of cycle on modern machines), you can hope that is it not too 
much of a problem (at least for memory allocation; I could not find in 
Andrew's slides whether he disabled the GC). But I doubt you can do much 
better.

I wonder if python can be compiled with a real time memory allocator, 
and even whether it makes sense at all (I am thinking about something 
like TLSF: http://rtportal.upv.es/rtmalloc/).
>  
> I hope that's changed, or is changing. I'd love to switch over to 64-bit 
> linux. As far as windows is considered, I'd like 32bit winxp to be my 
> last iteration.
With recent kernels, you can get really good latency if you do it right 
(around 1-2 ms worst case under high load, including high IO pressure). 
I know nothing about video programming, but I would guess that as far as 
the kernel is concerned, this does not change much. I have not tried 
them myself, but ubuntu studio has its own kernel with 'real-time' 
patched (voluntary preempt from Ingo, for example), and is available 
both for 32 and 64 bits architectures. One problem I can think of for 
video is that if you need binary-only drivers: those are generally 
pretty bad as far as low latency is concerned (nvidia drivers always 
cause some kind of problems with low latency and 'real-time' kernels).

http://ubuntustudio.org/

David