Multiprocessing bug, is my editor (SciTE) impeding my progress?

[Marco Nawijn]

On Dec 6, 8:13 pm, John Ladasky <lada... at my-deja.com> wrote:
> Hi, folks,
>
> Back in 2002, I got back into programming after a nine-year hiatus.  I
> needed a new programming language, was guided to Python 2.2, and was
> off to the races.  I chose the SciTE program editor, and I have been
> using it ever since.  I'm now using Python 2.6 on Ubuntu Linux 10.10.
>
> My programming needs have grown more sophisticated, but I'm still
> using SciTE.  Pretty much all of my recent posts to comp.lang.python
> have concerned multiprocessing.  I put together a decent system for my
> current project, and had it all working.  Then I realized that I
> needed to refactor and expand some code, which I did -- and somehow, I
> generated a bug that I simply cannot understand.  I've been puzzling
> over it for three days.
>
> The error is occurring inside one of my subprocesses.  As far as I
> know, SciTE is limited in what it can do in this situation.  The
> program does not return when a subprocess generates an exception.  I
> see the error message, but then the program simply hangs.
>
> I have tried invoking the subprocess directly without scheduling it
> through multiprocessing.Pool.  It works fine.  So the problem is
> occurring inside Pool.
>
> I tried opening my code in IDLE, and figured I could step through it,
> or at least call functions one line at a time.  It appears that
> multiprocessing code is not compatible with IDLE.  IDLE simply crashes
> when I try to invoke any of the important functions.
>
> I know, you want me to post a minimal example.  Most of the time,
> that's possible, and I do it.  Trust me, this time it isn't.  I have
> about 500 lines of code, split across three files.  These implement a
> neural network, some test data, and multiprocessing methods for
> network evaluation.  I made several concerted changes to the code, and
> turned a working system into this:
>
> =============================================
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/threading.py", line 532, in
> __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.6/threading.py", line 484, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> _handle_tasks
>     put(task)
> TypeError: expected string or Unicode object, NoneType found
>
> =============================================
>
> Here's what I think would help me debug this error: I would like to
> catch the TypeError, and examine the contents of task.  I need to
> accomplish this WITHOUT adding a try...except block to the Python
> library file multiprocessing/pool.py.  I don't know whether this is
> possible, because the traceback isn't clear about where my OWN code
> calls the code which is generating the error.
>
> After that, if the cause of the error still is not obvious, I might
> need to go back to the working program.  Somehow I want to examine the
> contents of task when the program works, and no TypeError is being
> generated.  By comparing the two, I hope to see a difference.  From
> that, I should be able to figure out how I have broken what is being
> fed to Pool.__init__ and/or MapResult.__init__.
>
> Any suggestions how I might best accomplish this task?  Does this
> error message look familiar to anyone?
>
> More generally, should I consider graduating from SciTE?  I have had a
> look at a few of the more comprehensive IDE's over the years, and I'll
> have to say that I found them to be intimidating.  I found it to be a
> huge chore just to open a single Python script and run it inside an
> IDE.  It seems like you had to know how to set up a complete, multi-
> script project before you could even accomplish simple tasks.  That
> steep learning curve is the reason that I didn't choose Java as my
> programming language.
>
> So, if any of you have pertinent recommendations in the IDE
> department, please feel free to guide me that way.
>
> Thanks!
Hello John,

One way of trying to debug the issue could be to use ipython and ipdb.
You cadn than run your code from within the ipython shell. The
debugger will hold at the type error, but keep the context. At this
point you should be able to evaluate task.

As a side comment to your IDE remarks. I keep switching between VIM
and Aptana/Pydev. The more I learn about VIM the more I feel
comfortable and productive. In combination with ipython it is quite a
solid development environment. On the other hand Pydev is very user
friendly, powerfull and easy to learn. Debugging in Pydev is
excellent.

Regards,

Marco