fromfile and tofile access with a tempfile.TemporaryFile()
Hi, Does anyone know how to get fromfile and tofile to work from a tempfile.TemporaryFile? Or if its not possible? I am getting this:
import tempfile f = tempfile.TemporaryFile() f <open file '<fdopen>', mode 'w+b' at 0x01EE1728> a = numpy.arange(10) a.tofile(f) Traceback (most recent call last): File "<input>", line 1, in ? IOError: first argument must be a string or open file
thanks! tim
On 12/11/06, Tim Hirzel <hirzel@resonon.com> wrote:
Hi, Does anyone know how to get fromfile and tofile to work from a tempfile.TemporaryFile? Or if its not possible?
I am getting this:
import tempfile f = tempfile.TemporaryFile() f <open file '<fdopen>', mode 'w+b' at 0x01EE1728> a = numpy.arange(10) a.tofile(f) Traceback (most recent call last): File "<input>", line 1, in ? IOError: first argument must be a string or open file
Works for me: In [16]: f = tempfile.TemporaryFile() In [17]: a = ones(10) In [18]: a.tofile(f) In [19]: f.seek(0) In [20]: b = fromfile(f) In [21]: b Out[21]: array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]) In [22]: f.close() What version of numpy are you running? Chuck
Hi Chuck, Thanks for checking that. I am running numpy 1.0.1 in python 2.4 on win32 (xp). Are you on linux? I double checked the behavior in 1.0 and 1.0.1, just to be extra sure, and it thows the IOError in both cases. tim Charles R Harris wrote:
On 12/11/06, *Tim Hirzel* <hirzel@resonon.com <mailto:hirzel@resonon.com>> wrote:
Hi, Does anyone know how to get fromfile and tofile to work from a tempfile.TemporaryFile? Or if its not possible?
I am getting this: >>> import tempfile >>> f = tempfile.TemporaryFile () >>> f <open file '<fdopen>', mode 'w+b' at 0x01EE1728> >>> a = numpy.arange(10) >>> a.tofile(f) Traceback (most recent call last): File "<input>", line 1, in ? IOError: first argument must be a string or open file
Works for me:
In [16]: f = tempfile.TemporaryFile()
In [17]: a = ones(10)
In [18]: a.tofile(f)
In [19]: f.seek(0)
In [20]: b = fromfile(f)
In [21]: b Out[21]: array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
In [22]: f.close()
What version of numpy are you running?
Chuck
------------------------------------------------------------------------
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On 12/12/06, Tim Hirzel <hirzel@resonon.com> wrote:
Hi Chuck, Thanks for checking that. I am running numpy 1.0.1 in python 2.4 on win32 (xp). Are you on linux? I double checked the behavior in 1.0 and 1.0.1, just to be extra sure, and it thows the IOError in both cases. tim
I'm running linux and the current svn version of numpy. Maybe the problem is with the tempfile module on windows. Do fromfile and tofile work for files opened normally? Chuck
I'm running linux and the current svn version of numpy. Maybe the problem is with the tempfile module on windows. Do fromfile and tofile work for files opened normally?
Chuck
fromfile and tofile work fine on regular files. From skimming the code a bit, it's hard to imagine numpy code is the culprit, since it must be getting a NULL pointer back from PyFile_AsFile(file)... Perhaps this is a question for a python dev list? My gut says it's probably something in the windows tempfile module. But perhaps in the PyFile_AsFile(file) implementation. Seems one of those isn't playing nice. It's all quite mysterious to me... tim
did you try reading and writing to/from that temp file with regular old python functions? -Chris Tim Hirzel wrote:
I'm running linux and the current svn version of numpy. Maybe the problem is with the tempfile module on windows. Do fromfile and tofile work for files opened normally?
Chuck
fromfile and tofile work fine on regular files. From skimming the code a bit, it's hard to imagine numpy code is the culprit, since it must be getting a NULL pointer back from PyFile_AsFile(file)... Perhaps this is a question for a python dev list? My gut says it's probably something in the windows tempfile module. But perhaps in the PyFile_AsFile(file) implementation. Seems one of those isn't playing nice. It's all quite mysterious to me...
tim
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Good thought Chris, Normal reading and writing does seem to work. .. But, my friend Daniel figured out a workaround when I asked to confirm this behavior on his windows setup (and it is does behave the same for him). The first clue was this:
f = tempfile.TemporaryFile() type(f) <type 'instance'> g = open("temp","w+b") type(g) <type 'file'>
so Daniel did a dir(f) and found the 'file' attribute so if you do (where 'a' is a numpy array)
a.tofile(f.file) It works!
writing to the "file" attribute of the TemporaryFile, it works fine! So that's good, but still a little hinky. Especially since it works on linux... on a linux platform, what does type(tempfile.TemporaryFile()) return? I assume an <type 'instance'> as well... anways, so at least there is a quick repair for now. Good news is, I assume using 'f.file' would work on linux too in terms of having a single cross-platform solution. cheers, tim Christopher Barker wrote:
did you try reading and writing to/from that temp file with regular old python functions?
-Chris
Tim Hirzel wrote:
I'm running linux and the current svn version of numpy. Maybe the problem is with the tempfile module on windows. Do fromfile and tofile work for files opened normally?
Chuck
fromfile and tofile work fine on regular files. From skimming the code a bit, it's hard to imagine numpy code is the culprit, since it must be getting a NULL pointer back from PyFile_AsFile(file)... Perhaps this is a question for a python dev list? My gut says it's probably something in the windows tempfile module. But perhaps in the PyFile_AsFile(file) implementation. Seems one of those isn't playing nice. It's all quite mysterious to me...
tim
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Tim Hirzel wrote:
Good thought Chris, Normal reading and writing does seem to work. .. But, my friend Daniel figured out a workaround when I asked to confirm this behavior on his windows setup (and it is does behave the same for him). The first clue was this:
f = tempfile.TemporaryFile() type(f) <type 'instance'> g = open("temp","w+b") type(g) <type 'file'>
so Daniel did a dir(f) and found the 'file' attribute
so if you do (where 'a' is a numpy array)
a.tofile(f.file) It works!
writing to the "file" attribute of the TemporaryFile, it works fine! So that's good, but still a little hinky. Especially since it works on linux... on a linux platform, what does type(tempfile.TemporaryFile()) return? I assume an <type 'instance'> as well...
anways, so at least there is a quick repair for now. Good news is, I assume using 'f.file' would work on linux too in terms of having a single cross-platform solution.
There is no file attribute on Linux. On linux you get
type(f) <type 'file'>
So, you might have to do something like: if not isinstance(f, file): f = f.file before passing f to the tofile method. It seems to me that the temporary file mechanism on Windows is a little odd. -Travis
It seems to me that the temporary file mechanism on Windows is a little odd.
Indeed, looking at sources, the posix version uses the mkstemp/unlink idiom.. but in win it uses a bit of hackery. It seems opened files cannot be unlinked. if _os.name != 'posix' or _os.sys.platform == 'cygwin': # On non-POSIX and Cygwin systems, assume that we cannot unlink a file # while it is open. TemporaryFile = NamedTemporaryFile else: def TemporaryFile(mode='w+b', bufsize=-1, suffix="", prefix=template, dir=None): ............ (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) try: _os.unlink(name) return _os.fdopen(fd, mode, bufsize) except: _os.close(fd) raise -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
On 12/12/06, Travis Oliphant <oliphant@ee.byu.edu> wrote:
Tim Hirzel wrote:
Good thought Chris, Normal reading and writing does seem to work. .. But, my friend Daniel figured out a workaround when I asked to confirm this behavior on his windows setup (and it is does behave the same for him). The first clue was this:
f = tempfile.TemporaryFile() type(f) <type 'instance'> g = open("temp","w+b") type(g) <type 'file'>
so Daniel did a dir(f) and found the 'file' attribute
so if you do (where 'a' is a numpy array)
a.tofile(f.file) It works!
writing to the "file" attribute of the TemporaryFile, it works fine! So that's good, but still a little hinky. Especially since it works on linux... on a linux platform, what does type(tempfile.TemporaryFile()) return? I assume an <type 'instance'> as well...
anways, so at least there is a quick repair for now. Good news is, I assume using 'f.file' would work on linux too in terms of having a single cross-platform solution.
There is no file attribute on Linux. On linux you get
type(f) <type 'file'>
So, you might have to do something like:
if not isinstance(f, file): f = f.file
before passing f to the tofile method.
It seems to me that the temporary file mechanism on Windows is a little odd.
Looks like a tempfile bug to me. Python should be cross platform and since the file attribute is correctly set on windows, I don't see why the tempfile can't be made to behave correctly. Chuck
Thanks for everyone's help and thoughts. I agree that this behavior is buggy. I submitted a bug report to the python project at sourceforge, with a link to this thread. Hopefully the report will be helpful. tim Charles R Harris wrote:
On 12/12/06, *Travis Oliphant* <oliphant@ee.byu.edu <mailto:oliphant@ee.byu.edu>> wrote:
Tim Hirzel wrote:
>Good thought Chris, >Normal reading and writing does seem to work. .. >But, my friend Daniel figured out a workaround when I asked to confirm >this behavior on his windows setup (and it is does behave the same for >him). The first clue was this: > > >>> f = tempfile.TemporaryFile() > >>> type(f) ><type 'instance'> > >>> g = open("temp","w+b") > >>> type(g) ><type 'file'> > >so Daniel did a dir(f) and found the 'file' attribute > >so if you do (where 'a' is a numpy array) > >>> a.tofile(f.file) >It works! > >writing to the "file" attribute of the TemporaryFile, it works fine! So >that's good, but still a little hinky. Especially since it works on >linux... >on a linux platform, what does type( tempfile.TemporaryFile()) return? I >assume an <type 'instance'> as well... > >anways, so at least there is a quick repair for now. Good news is, I >assume using 'f.file' would work on linux too in terms of having a >single cross-platform solution. > > There is no file attribute on Linux. On linux you get
>>> type(f) <type 'file'>
So, you might have to do something like:
if not isinstance(f, file): f = f.file
before passing f to the tofile method.
It seems to me that the temporary file mechanism on Windows is a little odd.
Looks like a tempfile bug to me. Python should be cross platform and since the file attribute is correctly set on windows, I don't see why the tempfile can't be made to behave correctly.
Chuck
------------------------------------------------------------------------
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
participants (5)
-
Charles R Harris -
Christopher Barker -
Lisandro Dalcin -
Tim Hirzel -
Travis Oliphant