Long strings as function parameters

[Jeremy Bowers]

On Sun, 09 Jan 2005 08:36:04 -0800, onlyonemc wrote:

> I would like to have functions that operate on long strings, 10-100 MB. In
> C I would of course pass a pointer to the string for a quick function
> call.  What is an efficient way to do this in python? Cheers,
> -mark


Others have pointed out that there is no particular inefficient way to
pass a string.

However that only solves the immediate problem of passing. Once inside the
function you should be aware that all string operations that change the
string will create a new string, including slicing and such.

This may not be a problem if you are examining it in small chunks, but if
you routinely taking distinct multi-megabyte hunks out of it with slices,
you will be constructing and destructing a lot of strings.

I had thought there was an obvious class in the standard library to assist
with this, but I must have been wrong. I think you can load it as an array
from the array module (as long as it is an array of bytes and not an
encoding like UTF-8 or something), or you might be able to use the mmap
module if the string comes from a file. Both of those techniques can also
load the info directly from a file so there is only one copy needed.
(Wasn't there a "buffer" kind of class at one point, that you could slice
into and get something back that didn't make any copies of strings?)

Again, this is only an issue depending on usage, and you should probably
prototype it while ignoring these issues and see if it is fast enough. But
if it isn't, there are options.