[issue9942] Allow memory sections to be OS MERGEABLE

Kevin Hunter report at bugs.python.org
Sat Sep 25 16:26:04 CEST 2010


Kevin Hunter <hunteke at earlham.edu> added the comment:

> Well, first, this would only work for large objects. [...]
> Why do you think you might have such duplication in your workload?

Some of the projects with which I work involve multiple manipulations of large datasets.  Often, we use Python scripts as "first and third" stages in a pipeline.  For example, in one current workflow, we read a large file into a cStringIO object, do a few manipulations with it, pass it off to a second process, and await the results.  Meanwhile, the large file is sitting around in memory because we need to do more manipulations after we get results back from the second application in the pipeline.  "Graphically":

Python Script A    ->    External App    ->    Python Script A
read large data          process data          more manipulations

Within a single process, I don't see any gain to be had.  However, in this one use-case, this pipeline is running concurrently with a number of copies with slightly different command line parameters.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9942>
_______________________________________


More information about the Python-bugs-list mailing list