I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap. It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
On Fri, Jan 9, 2009 at 06:05, Neal Becker
I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap.
It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
Use os.fdopen(fd) to create a file object which can be passed to the memmap constructor. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
On Fri, Jan 9, 2009 at 06:05, Neal Becker
wrote: I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap.
It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
Use os.fdopen(fd) to create a file object which can be passed to the memmap constructor.
Thanks! I'm assuming in this case I can ignore comments about flushing the data to disk? If an assignment to a slice of an memmap array will call mmap, there should be no need for any flushing.
Robert Kern wrote:
On Fri, Jan 9, 2009 at 06:05, Neal Becker
wrote: I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap.
It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
Use os.fdopen(fd) to create a file object which can be passed to the memmap constructor.
Looks like this is not going to work without some change to memmap. The problem is, I need read/write access. The only choice in memmap is 'w+'. But this does: if (mode == 'w+') and shape is None: raise ValueError, "shape must be given" fid.seek(0,2) My device has hijacked 'read' to mean something entirely different than you might expect. The seek call invokes 'read'. It looks like the purpose of this code is to find the size of the mappable area. The best solution I think is just throw it away. Consistent with mmap semantics, attempting access outside the mappable area should cause and error - but I don't think there is any reliable way to know the length of the mappable area apriori. Any thoughts?
On Fri, Jan 9, 2009 at 08:08, Neal Becker
Robert Kern wrote:
On Fri, Jan 9, 2009 at 06:05, Neal Becker
wrote: I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap.
It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
Use os.fdopen(fd) to create a file object which can be passed to the memmap constructor.
Looks like this is not going to work without some change to memmap. The problem is, I need read/write access. The only choice in memmap is 'w+'.
'r+' is for reading and writing.
But this does: if (mode == 'w+') and shape is None: raise ValueError, "shape must be given"
fid.seek(0,2)
My device has hijacked 'read' to mean something entirely different than you might expect. The seek call invokes 'read'.
It looks like the purpose of this code is to find the size of the mappable area. The best solution I think is just throw it away.
We can't. We need it.
Consistent with mmap semantics, attempting access outside the mappable area should cause and error - but I don't think there is any reliable way to know the length of the mappable area apriori.
For regular files, that seems to me to be fairly reliable. Why isn't it? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
On Fri, Jan 9, 2009 at 08:08, Neal Becker
wrote: Robert Kern wrote:
On Fri, Jan 9, 2009 at 06:05, Neal Becker
wrote: I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap.
It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
Use os.fdopen(fd) to create a file object which can be passed to the memmap constructor.
Looks like this is not going to work without some change to memmap. The problem is, I need read/write access. The only choice in memmap is 'w+'.
'r+' is for reading and writing.
But this does: if (mode == 'w+') and shape is None: raise ValueError, "shape must be given"
fid.seek(0,2)
My device has hijacked 'read' to mean something entirely different than you might expect. The seek call invokes 'read'.
It looks like the purpose of this code is to find the size of the mappable area. The best solution I think is just throw it away.
We can't. We need it.
Consistent with mmap semantics, attempting access outside the mappable area should cause and error - but I don't think there is any reliable way to know the length of the mappable area apriori.
For regular files, that seems to me to be fairly reliable. Why isn't it?
Because I'm not mmapping a file. I'm mmapping a device. It exposes the memory of the FPGA board as seen on the PCI bus. You just have to know the size.
On Fri, Jan 9, 2009 at 22:15, Neal Becker
Robert Kern wrote:
On Fri, Jan 9, 2009 at 08:08, Neal Becker
wrote: Robert Kern wrote:
On Fri, Jan 9, 2009 at 06:05, Neal Becker
wrote: I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap.
It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?
Use os.fdopen(fd) to create a file object which can be passed to the memmap constructor.
Looks like this is not going to work without some change to memmap. The problem is, I need read/write access. The only choice in memmap is 'w+'.
'r+' is for reading and writing.
But this does: if (mode == 'w+') and shape is None: raise ValueError, "shape must be given"
fid.seek(0,2)
My device has hijacked 'read' to mean something entirely different than you might expect. The seek call invokes 'read'.
It looks like the purpose of this code is to find the size of the mappable area. The best solution I think is just throw it away.
We can't. We need it.
Consistent with mmap semantics, attempting access outside the mappable area should cause and error - but I don't think there is any reliable way to know the length of the mappable area apriori.
For regular files, that seems to me to be fairly reliable. Why isn't it?
Because I'm not mmapping a file. I'm mmapping a device. It exposes the memory of the FPGA board as seen on the PCI bus. You just have to know the size.
That's a fair point. However, given the wider range of issues that you had in trying to get memmap to work with your device, perhaps we should just state that the memmap class is intended for the common use case of regular files. The detection of the existing file size is quite useful in that case. For special files where seek() et al. might have different semantics, one should either subclass memmap to get the behavior you need, or simply use frombuffer() or ndarray() on the mmap object directly. The two reasons for the memmap class are the auto-length-detection and tracking the lifetime of the open file. The former you don't need (and can't do), and the latter isn't that hard to do yourself. Does that sound reasonable? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote: ....
That's a fair point. However, given the wider range of issues that you had in trying to get memmap to work with your device, perhaps we should just state that the memmap class is intended for the common use case of regular files. The detection of the existing file size is quite useful in that case. For special files where seek() et al. might have different semantics, one should either subclass memmap to get the behavior you need, or simply use frombuffer() or ndarray() on the mmap object directly. The two reasons for the memmap class are the auto-length-detection and tracking the lifetime of the open file. The former you don't need (and can't do), and the latter isn't that hard to do yourself.
Does that sound reasonable?
Hmm. frombuffer sounds nice, but python mmap doesn't expose buffer interface (I just added a feature request for this). I suppose I could write my own mmap module. I couldn't find the reference to ndarray(). Any pointer?
On Sun, Jan 11, 2009 at 20:23, Neal Becker
Robert Kern wrote:
....
That's a fair point. However, given the wider range of issues that you had in trying to get memmap to work with your device, perhaps we should just state that the memmap class is intended for the common use case of regular files. The detection of the existing file size is quite useful in that case. For special files where seek() et al. might have different semantics, one should either subclass memmap to get the behavior you need, or simply use frombuffer() or ndarray() on the mmap object directly. The two reasons for the memmap class are the auto-length-detection and tracking the lifetime of the open file. The former you don't need (and can't do), and the latter isn't that hard to do yourself.
Does that sound reasonable?
Hmm. frombuffer sounds nice, but python mmap doesn't expose buffer interface (I just added a feature request for this). I suppose I could write my own mmap module.
Yeah, it does. Why do you think it doesn't? If it didn't work, the memmap class wouldn't work.
I couldn't find the reference to ndarray(). Any pointer?
http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html The buffer= argument would be the mmap object. Basically, most of what memmap.__new__() does is to set up the arguments for ndarray.__new__(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote: ...
Hmm. frombuffer sounds nice, but python mmap doesn't expose buffer interface (I just added a feature request for this). I suppose I could write my own mmap module.
Yeah, it does. Why do you think it doesn't? If it didn't work, the memmap class wouldn't work.
My mistake.
participants (2)
-
Neal Becker
-
Robert Kern