[Numpy-discussion] out parameter for np.fromfile

Mark Harfouche mark.harfouche at gmail.com
Mon Nov 5 11:24:57 EST 2018


Thanks Marten.

I tried memap for a few things but it seemed to create an other OS level
buffer in specific situations. I think the `seek` operation in the `memmap`
also caused some performance bottlenecks. Maybe I'll have time to summarize
my findings an other day.

The particular usecase of `ffmpeg` is tricky since it is grabbing a lot of
data from `stdout` which isn't a typical file buffer. Specifically, it is
often `buffered` but `unseekable`.

On Mon, Nov 5, 2018 at 9:38 AM Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> Hi Mark,
>
> Having an `out` might make sense. With present numpy, if you are really
> dealing with a file or file-like object, you might consider using
> `np.memmap` to access the data more directly. If it is something that looks
> more like a buffer, `np.frombuffer` may be useful (that doesn't copy data,
> but points the array at the memory that holds the buffer).
>
> All the best,
>
> Marten
>
>
> On Sun, Nov 4, 2018 at 10:35 PM Mark Harfouche <mark.harfouche at gmail.com>
> wrote:
>
>> I was wondering what would your thoughts be on adding an output parameter
>> to np.fromfile?
>>
>> The advantage would be when interfacing with executables like ffmpeg
>> which are arguably easier to use by calling them as a subprocess compared
>> to a shared library in python.
>>
>> Having the output parameter in np.fromfile would enable pre-allocation
>> of large arrays that are reused during the computation of new image frames
>> when decoding large video files.
>>
>> Thoughts are appreciated!
>>
>> Mark
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20181105/d8d39ba5/attachment-0001.html>


More information about the NumPy-Discussion mailing list