2.6, 3.0, and truly independent intepreters
v+python at g.nevcal.com
Thu Oct 30 23:54:29 CET 2008
On approximately 10/30/2008 6:26 AM, came the following characters from
the keyboard of Jesse Noller:
> On Wed, Oct 29, 2008 at 8:05 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
>> On approximately 10/29/2008 3:45 PM, came the following characters from the
>> keyboard of Patrick Stinson:
>>> If you are dealing with "lots" of data like in video or sound editing,
>>> you would just keep the data in shared memory and send the reference
>>> over IPC to the worker process. Otherwise, if you marshal and send you
>>> are looking at a temporary doubling of the memory footprint of your
>>> app because the data will be copied, and marshaling overhead.
>> Right. Sounds, and is, easy, if the data is all directly allocated by the
>> application. But when pieces are allocated by 3rd party libraries, that use
>> the C-runtime allocator directly, then it becomes more difficult to keep
>> everything in shared memory.
>> One _could_ replace the C-runtime allocator, I suppose, but that could have
>> some adverse effects on other code, that doesn't need its data to be in
>> shared memory. So it is somewhat between a rock and a hard place.
>> By avoiding shared memory, such problems are sidestepped... until you run
>> smack into the GIL.
> If you do not have shared memory: You don't need threads, ergo: You
> don't get penalized by the GIL. Threads are only useful when you need
> to have that requirement of large in-memory data structures shared and
> modified by a pool of workers.
The whole point of this thread is to talk about large in-memory data
structures that are shared and modified by a pool of workers.
My reference to shared memory was specifically referring to the concept
of sharing memory between processes... a particular OS feature that is
called shared memory.
The need for sharing memory among a pool of workers is still the
premise. Threads do that automatically, without the need for the OS
shared memory feature, that brings with it the need for a special
allocator to allocate memory in the shared memory area vs the rest of
the address space.
Not to pick on you, particularly, Jesse, but this particular response
made me finally understand why there has been so much repetition of the
same issues and positions over and over and over in this thread: instead
of comprehending the whole issue, people are responding to small
fragments of it, with opinions that may be perfectly reasonable for that
fragment, but missing the big picture, or the explanation made when the
same issue was raised in a different sub-thread.
Glenn -- http://nevcal.com/
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
More information about the Python-list