How to write fast into a file in python?
Carlos Nepomuceno
carlosnepomuceno at outlook.com
Sun May 19 01:31:08 EDT 2013
Thanks Dan! I've never used CPython or PyPy. Will try them later.
I think the main difference between your create_file_numbers_file_like()
and the fastwrite5.py I sent earlier is that I've used cStringIO
instead of StringIO. It took 12s less using cStringIO.
My numbers are much greater, but I've used Python 2.7.5 instead:
C:\src\Python>python create_file_numbers.py
time taken to write a file of size 52428800 is 39.1199457743 seconds
time taken to write a file of size 52428800 is 14.8704800436 seconds
time taken to write a file of size 52428800 is 23.0011990985 seconds
I've downloaded bufsock.py and python2x3.py. The later one was hard to remove the source code from the web page.
Can I use them on my projects? I'm not used to the UCI license[1]. What's the difference to the GPL?
[1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html
________________________________
> Date: Sat, 18 May 2013 12:38:30 -0700
> Subject: Re: How to write fast into a file in python?
> From: drsalists at gmail.com
> To: lokeshkoppaka at gmail.com
> CC: python-list at python.org
>
>
> With CPython 2.7.3:
> ./t
> time taken to write a file of size 52428800 is 15.86 seconds
>
> time taken to write a file of size 52428800 is 7.91 seconds
>
> time taken to write a file of size 52428800 is 9.64 seconds
>
>
> With pypy-1.9:
> ./t
> time taken to write a file of size 52428800 is 3.708232 seconds
>
> time taken to write a file of size 52428800 is 4.868304 seconds
>
> time taken to write a file of size 52428800 is 1.93612 seconds
>
> Here's the code:
> #!/usr/local/pypy-1.9/bin/pypy
> #!/usr/bin/python
>
> import sys
> import time
> import StringIO
>
> sys.path.insert(0, '/usr/local/lib')
> import bufsock
>
> def create_file_numbers_old(filename, size):
> start = time.clock()
>
> value = 0
> with open(filename, "w") as f:
> while f.tell() < size:
> f.write("{0}\n".format(value))
> value += 1
>
> end = time.clock()
>
> print "time taken to write a file of size", size, " is ", (end
> -start), "seconds \n"
>
> def create_file_numbers_bufsock(filename, intended_size):
> start = time.clock()
>
> value = 0
> with open(filename, "w") as f:
> bs = bufsock.bufsock(f)
> actual_size = 0
> while actual_size < intended_size:
> string = "{0}\n".format(value)
> actual_size += len(string) + 1
> bs.write(string)
> value += 1
> bs.flush()
>
> end = time.clock()
>
> print "time taken to write a file of size", intended_size, " is ",
> (end -start), "seconds \n"
>
>
> def create_file_numbers_file_like(filename, intended_size):
> start = time.clock()
>
> value = 0
> with open(filename, "w") as f:
> file_like = StringIO.StringIO()
> actual_size = 0
> while actual_size < intended_size:
> string = "{0}\n".format(value)
> actual_size += len(string) + 1
> file_like.write(string)
> value += 1
> file_like.seek(0)
> f.write(file_like.read())
>
> end = time.clock()
>
> print "time taken to write a file of size", intended_size, " is ",
> (end -start), "seconds \n"
>
> create_file_numbers_old('output.txt', 50 * 2**20)
> create_file_numbers_bufsock('output2.txt', 50 * 2**20)
> create_file_numbers_file_like('output3.txt', 50 * 2**20)
>
>
>
>
> On Thu, May 16, 2013 at 9:35 PM,
> <lokeshkoppaka at gmail.com<mailto:lokeshkoppaka at gmail.com>> wrote:
> On Friday, May 17, 2013 8:50:26 AM UTC+5:30,
> lokesh... at gmail.com<mailto:lokesh... at gmail.com> wrote:
> > I need to write numbers into a file upto 50mb and it should be fast
> >
> > can any one help me how to do that?
> >
> > i had written the following code..
> >
> >
> -----------------------------------------------------------------------------------------------------------
> >
> > def create_file_numbers_old(filename, size):
> >
> > start = time.clock()
> >
> >
> >
> > value = 0
> >
> > with open(filename, "w") as f:
> >
> > while f.tell()< size:
> >
> > f.write("{0}\n".format(value))
> >
> > value += 1
> >
> >
> >
> > end = time.clock()
> >
> >
> >
> > print "time taken to write a file of size", size, " is ", (end
> -start), "seconds \n"
> >
> >
> ------------------------------------------------------------------------------------------------------------------
> >
> > it takes about 20sec i need 5 to 10 times less than that.
> size = 50mb
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
> -- http://mail.python.org/mailman/listinfo/python-list
More information about the Python-list
mailing list