[py-dev] execnet: shutdown issues in a heavily threaded environment

[Gordon Wrigley]

>> Also this only started when I reworked our remote invocation system
>> recently, the most notable change in the new system is that it makes
>> more use of threads and particularly it has daemon threads waiting on
>> channel receive calls.

> Might also be python-version dependent.  What are
> the differences between your test and production system?

development machine:

gordonw at gohma:~$ uname -a
Linux gohma 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC
2009 i686 GNU/Linux
gordonw at gohma:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b7'

test machine:

lab at okum:~$ uname -a
Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
2009 i686 GNU/Linux
lab at okum:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import py
>>> py.version
'1.0.0b1'

On Tue, Jul 14, 2009 at 7:10 PM, holger krekel<holger at merlinux.eu> wrote:
> Hi Gordon,
>
> On Mon, Jul 13, 2009 at 11:40 +1000, Gordon Wrigley wrote:
>> > One issue that can pop up is tearing down the whole
>> > machinery cleanly - that can sometimes be more
>> > involved and i believe execnet still might need
>> > some more work there.
>>
>> On that note can you shed any light on this message:
>>
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel._del_ of <Channel id=7
>> open>> ignored
>>
>> Currently I'm only seeing them on my test machine, but they are
>> occurring very frequently, they drop out at the end of the py.test
>> invocation, I will paste the tail end of the py.test output below.
>
>> I'm not sure exactly what is causing this, naively it seems to
>> indicate a circular chain of references involving one or more
>> channels. However immediately prior to the conclusion of each test
>> exit gc.garbage is empty and there are none of my objects in
>> gc.get_objects.
>
> Yes, some cycle is probably there.  Your data point hints that
> there is some cycle in the execnet code itself.
>
>> Also this only started when I reworked our remote invocation system
>> recently, the most notable change in the new system is that it makes
>> more use of threads and particularly it has daemon threads waiting on
>> channel receive calls.
>
> Hum, that might be related - GC-finalization issues in
> MT-environments are not easy to debug :(
> Might also be python-version dependent.  What are
> the differences between your test and production system?
>
> I am a bit at a loss on how to best proceed at the moment.
> I think that adding more debugging information
> to execnet and systematically implementing and
> checking scenarios is due - but quite a bit of effort.
> Could you maybe try coming up with an self-contained
> example test-script leading to these messages?
>
> On a sidenote, probably around 60% of the execnet core
> programming effort revolve around teardown/finalization issues
> - i wonder if it would be better to avoid usage of __del__
> alltogether, only have a process-wide atexit handler and
> otherwise recommend explicit calling of
> gateway.exit()/channel.close methods for proper resource
> handling.
>
> best,
>
> holger
>
>> lab at okum:~$ uname -a
>> Linux okum 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC
>> 2009 i686 GNU/Linux
>> lab at okum:~$ python
>> Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
>> [GCC 4.3.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import py
>> >>> py.version
>> '1.0.0b1'
>>
>>
>> ============================================== 35 passed in 231.71
>> seconds ===============================================
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in Unhandled exception in thread started by <bound
>> method Thread.__bootstrap of <Thread(receiver, stopped daemon
>> 1177635728)>>
>> <bound method Channel.__del__ of <Channel id=7 open>>Error in sys.excepthook:
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/dist-packages/apport_python_hook.py", line
>> 38, in apport_excepthook
>>     from apport.packaging_impl import impl as packaging
>> ImportError: No module named apport.packaging_impl
>>
>> Original exception was:
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
>>     self.__bootstrap_inner()
>>   File "/usr/lib/python2.6/threading.py", line 538, in __bootstrap_inner
>>     (self.name, _format_exc()))
>> TypeError: 'NoneType' object is not callable
>>  ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> Exception AttributeError: "'NoneType' object has no attribute
>> 'CHANNEL_CLOSE'" in <bound method Channel.__del__ of <Channel id=7
>> open>> ignored
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>
>
> --
> Metaprogramming, Python, Testing: http://tetamap.wordpress.com
> Python, PyPy, pytest contracting: http://merlinux.eu
>