Multiprocessing bug, is my editor (SciTE) impeding my progress?

[John Ladasky]

On Dec 6, 1:42 pm, Terry Reedy <tjre... at udel.edu> wrote:
> On 12/6/2011 2:13 PM, John Ladasky wrote:
> > Exception in thread Thread-1:
> > Traceback (most recent call last):
> >    File "/usr/lib/python2.6/threading.py", line 532, in
> > __bootstrap_inner
> >      self.run()
> >    File "/usr/lib/python2.6/threading.py", line 484, in run
> >      self.__target(*self.__args, **self.__kwargs)
> >    File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> > _handle_tasks
> >      put(task)
> > TypeError: expected string or Unicode object, NoneType found
>
> > =============================================
>
> > Here's what I think would help me debug this error: I would like to
> > catch the TypeError, and examine the contents of task.
>
> The traceback says that it is None, which has no contents ;=).

I'm not sure about that.  I don't submit the task variable, it's
something that Pool builds from what I submit.  Is task == None when
it should be a string?  Or, is task an iterable which contains one
element which should be a string?  And what's supposed to be in that
string anyway?  My first reading of the source code of Pool didn't
make this clear to me.  Also, I've noticed that tracebacks from
subprocesses are less informative than tracebacks from the parent
process.  What's missing?

Finally, I also recall that multiprocessing invokes pickle to pass
data between processes (I still don't understand the need for pickle
here).  This suggests to me that the string MIGHT contain the pickled
code of the method I want to run in the subprocess.  But I'm not sure,
until I can actually examine the task variable in my working version.

> > I need to
> > accomplish this WITHOUT adding a try...except block to the Python
> > library file multiprocessing/pool.py.
>
> I do not understand this statement. You should feel free to make a
> backup copy of pool.py and then modify it for debugging.

Right, so, the last time I tried this with a piece of library code, I
ran into some major headaches with import statements.  I suppose I
could have a look at Pool and see whether it can be extracted cleanly
and made to run.

>  > I don't know whether this is
>
> > possible, because the traceback isn't clear about where my OWN code
> > calls the code which is generating the error.
>
> It appears to be threading trying to call your code, and failing, that
> is the problem. But I do not know how threading, multiprocessing.pool,
> and your code are supposed to interact.

It might be that pickle has somehow managed to pass a null code string
to the subprocess, so it has nothing to run.

> > After that, if the cause of the error still is not obvious, I might
> > need to go back to the working program.  Somehow I want to examine the
> > contents of task when the program works, and no TypeError is being
> > generated.
>
> The traceback says that it is a string.

Yes, again... I want to know what that string is supposed to DO.

> I would start with the line that fails 'put(task)', and work backwards
> to see where 'task' comes from and how it could become None. It is even
> possible that multiprocessing.pool has a bug that you ran into.

Oh, please don't say that.  I'm no computer scientist, and Python has
been scrutinized by so many professionals.  I couldn't have possibly
found a language bug.