[python-win32] Passing an object to a process

[Preston Landers]

On Wed, Mar 14, 2012 at 2:55 PM, Tony Cappellini <cappy2112 at gmail.com> wrote:
>
>> >>in which objects can truly move from one process to another is
>> >>recreating them in the other process.  Even fork() makes copies of
>> >> everything.
>
> Recreating an object in another process means it's a different object, not a
> shared one.

Yeah, I know.  I was trying to make that point.  There's no real way
for the same object to exist in multiple processes other than SYSV
shared mem.

Truly shared memory (i.e, SYSV style) is tricky, not very portable,
and usually the wrong answer in my experience. As fas as I know stock
Python doesn't support that, and definitely never will on Windows.

The point is that you need to figure out what problem you're really
trying to solve (logging to one file from multiple processes, it
sounds like) and then find the best / simplest approach, which I can
tell you definitely doesn't involve SYSV shared memory.   It's
probably just creating separate logging objects in each process,
pointing to the same file, and protected by file locking if necessary.


>> >>Have you tried pickle or other techniques of serialization? Not sure
>> >>offhand if the logger module supports pickle but it might.
>
> Yes. I've just tried this, even though I expected it not to work.
>
> If process A pickles a logger object, and process B unpickles it,
> referencing of an object in a different process is meaningless.
>
> In my case, when the process attempted to write to the logger, no entries
> were seen in the logfile. Surprisingly, no exceptions occurred- but this
> could just be a coincidence.

Probably because the logger object, when serialized, saves a reference
to an open filehandle, which won't be automatically transfered to the
other process.

(There might ultimately be a way to make that work by inheriting
filehandles, but again, if you can find something simpler...)


>
> That may work, and with less effort than my original idea.
>
> But if two processes write to the logfile at the same time (especially on a
> multicore machine),
> hard-to-read logfiles may result.
> it's worth a try.

Yes, that type of thing can occur, but you can also get around that
with simple file locking.  By the way, that same problem certainly
exists even if you somehow shared the object between two processes -
what if the two processes made a log call at the same time?

File locking may introduce some performance issues if the logging is
very frequent, but usually you can find ways to mitigate that.

The main app I work on uses log files extensively, and the same file
is appended to by unrelated processes without a locking mechanism.
Occasionally you do see some interleaving of log entries but in my
experience it's fairly rare and in my particular case we don't care
much about that anyway.

regards,
Preston