[python-win32] Passing an object to a process

[Max Slimmer]

If you are really just logging, use separate loggers/files, sync the
clocks, and timestamp records, then merge when you are ready to
process if you need timeline related results.


On Wed, Mar 14, 2012 at 1:29 PM, Preston Landers <planders at gmail.com> wrote:
> On Wed, Mar 14, 2012 at 2:55 PM, Tony Cappellini <cappy2112 at gmail.com> wrote:
>>
>>> >>in which objects can truly move from one process to another is
>>> >>recreating them in the other process.  Even fork() makes copies of
>>> >> everything.
>>
>> Recreating an object in another process means it's a different object, not a
>> shared one.
>
> Yeah, I know.  I was trying to make that point.  There's no real way
> for the same object to exist in multiple processes other than SYSV
> shared mem.
>
> Truly shared memory (i.e, SYSV style) is tricky, not very portable,
> and usually the wrong answer in my experience. As fas as I know stock
> Python doesn't support that, and definitely never will on Windows.
>
> The point is that you need to figure out what problem you're really
> trying to solve (logging to one file from multiple processes, it
> sounds like) and then find the best / simplest approach, which I can
> tell you definitely doesn't involve SYSV shared memory.   It's
> probably just creating separate logging objects in each process,
> pointing to the same file, and protected by file locking if necessary.
>
>
>>> >>Have you tried pickle or other techniques of serialization? Not sure
>>> >>offhand if the logger module supports pickle but it might.
>>
>> Yes. I've just tried this, even though I expected it not to work.
>>
>> If process A pickles a logger object, and process B unpickles it,
>> referencing of an object in a different process is meaningless.
>>
>> In my case, when the process attempted to write to the logger, no entries
>> were seen in the logfile. Surprisingly, no exceptions occurred- but this
>> could just be a coincidence.
>
> Probably because the logger object, when serialized, saves a reference
> to an open filehandle, which won't be automatically transfered to the
> other process.
>
> (There might ultimately be a way to make that work by inheriting
> filehandles, but again, if you can find something simpler...)
>
>
>>
>> That may work, and with less effort than my original idea.
>>
>> But if two processes write to the logfile at the same time (especially on a
>> multicore machine),
>> hard-to-read logfiles may result.
>> it's worth a try.
>
> Yes, that type of thing can occur, but you can also get around that
> with simple file locking.  By the way, that same problem certainly
> exists even if you somehow shared the object between two processes -
> what if the two processes made a log call at the same time?
>
> File locking may introduce some performance issues if the logging is
> very frequent, but usually you can find ways to mitigate that.
>
> The main app I work on uses log files extensively, and the same file
> is appended to by unrelated processes without a locking mechanism.
> Occasionally you do see some interleaving of log entries but in my
> experience it's fairly rare and in my particular case we don't care
> much about that anyway.
>
> regards,
> Preston
> _______________________________________________
> python-win32 mailing list
> python-win32 at python.org
> http://mail.python.org/mailman/listinfo/python-win32