[python-win32] Passing an object to a process

[Tony Cappellini]

Thanks for all the advice.

The solution is simple, but neither elegant nor Pythonic. It's a kludge,
but will suffice.

The process simply opens & writes a 1 line log file, then closes it. The
process also increments a counter
each time the log file is written, and writes the counter value to the
process log file.

The program checks to see if the file exists, reads it, then writes the
process log file content to the program log file.
The program then deletes the process log file. (probably unnecessary, but
prevents the possibility of the same line in the process log file
from being written to the program log file more than once.)

Performance isn't a concern, and this code won't be going to production.
It's just for debugging.

On Wed, Mar 14, 2012 at 1:49 PM, Max Slimmer <max at slimmersoft.com> wrote:

> If you are really just logging, use separate loggers/files, sync the
> clocks, and timestamp records, then merge when you are ready to
> process if you need timeline related results.
>
>
> On Wed, Mar 14, 2012 at 1:29 PM, Preston Landers <planders at gmail.com>
> wrote:
> > On Wed, Mar 14, 2012 at 2:55 PM, Tony Cappellini <cappy2112 at gmail.com>
> wrote:
> >>
> >>> >>in which objects can truly move from one process to another is
> >>> >>recreating them in the other process.  Even fork() makes copies of
> >>> >> everything.
> >>
> >> Recreating an object in another process means it's a different object,
> not a
> >> shared one.
> >
> > Yeah, I know.  I was trying to make that point.  There's no real way
> > for the same object to exist in multiple processes other than SYSV
> > shared mem.
> >
> > Truly shared memory (i.e, SYSV style) is tricky, not very portable,
> > and usually the wrong answer in my experience. As fas as I know stock
> > Python doesn't support that, and definitely never will on Windows.
> >
> > The point is that you need to figure out what problem you're really
> > trying to solve (logging to one file from multiple processes, it
> > sounds like) and then find the best / simplest approach, which I can
> > tell you definitely doesn't involve SYSV shared memory.   It's
> > probably just creating separate logging objects in each process,
> > pointing to the same file, and protected by file locking if necessary.
> >
> >
> >>> >>Have you tried pickle or other techniques of serialization? Not sure
> >>> >>offhand if the logger module supports pickle but it might.
> >>
> >> Yes. I've just tried this, even though I expected it not to work.
> >>
> >> If process A pickles a logger object, and process B unpickles it,
> >> referencing of an object in a different process is meaningless.
> >>
> >> In my case, when the process attempted to write to the logger, no
> entries
> >> were seen in the logfile. Surprisingly, no exceptions occurred- but this
> >> could just be a coincidence.
> >
> > Probably because the logger object, when serialized, saves a reference
> > to an open filehandle, which won't be automatically transfered to the
> > other process.
> >
> > (There might ultimately be a way to make that work by inheriting
> > filehandles, but again, if you can find something simpler...)
> >
> >
> >>
> >> That may work, and with less effort than my original idea.
> >>
> >> But if two processes write to the logfile at the same time (especially
> on a
> >> multicore machine),
> >> hard-to-read logfiles may result.
> >> it's worth a try.
> >
> > Yes, that type of thing can occur, but you can also get around that
> > with simple file locking.  By the way, that same problem certainly
> > exists even if you somehow shared the object between two processes -
> > what if the two processes made a log call at the same time?
> >
> > File locking may introduce some performance issues if the logging is
> > very frequent, but usually you can find ways to mitigate that.
> >
> > The main app I work on uses log files extensively, and the same file
> > is appended to by unrelated processes without a locking mechanism.
> > Occasionally you do see some interleaving of log entries but in my
> > experience it's fairly rare and in my particular case we don't care
> > much about that anyway.
> >
> > regards,
> > Preston
> > _______________________________________________
> > python-win32 mailing list
> > python-win32 at python.org
> > http://mail.python.org/mailman/listinfo/python-win32
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-win32/attachments/20120314/c7e2d506/attachment-0001.html>