<div dir="ltr">Thank you Andrew, good to know you could replicate it. I went to very similar workarround with __contains__(), but yours is more elegant with the "while 1". However, I will recommend to repeat only a few times (to avoid a possible endless loop) and to catch the two unexpected exception only <span style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px">(SystemError, ValueError)</span>. So I suggest the following workaround based in your proposal should be: <div><br></div><div><div style="font-family:Arial;font-size:13.3333330154419px"><span style="color:rgb(0,0,0)"> </span><font color="#0000ff"> <span style="font-size:13.3333330154419px">def __contains__(self, item):</span></font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> while 1:</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> try:</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> try:</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> wr = ref(item) </font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> except TypeError:</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> return False</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> return wr in self.data</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><font color="#0000ff"> except (SystemError, ValueError):</font><font color="#666666"> #i.e: Don't catch KeyboardInterrupt, MemoryError, etc</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><span style="color:rgb(0,0,0)"> </span><font color="#0000ff"> try: i+=1 </font></div><div style="font-family:Arial;font-size:13.3333330154419px"><span style="color:rgb(0,0,0)"> </span><font color="#0000ff">except: i=0 </font><span style="color:rgb(0,0,0)"> </span><font color="#666666"> # i declared here. Don't want waste resource outside... (not Pythonic, I know)</font></div><div style="font-family:Arial;font-size:13.3333330154419px"><span style="color:rgb(0,0,0)"> </span><font color="#0000ff">if i==10: raise</font><span style="color:rgb(0,0,0)"> </span><font color="#666666"> # Retry max 10 times and re-raise </font></div></div><div style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px"><br></div><div style><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px">However, would like if someone can debug the error in the C# code of weakref. I'm afraid that the workaround will leave some potential error open beside __contains__(). I see than inside </span></font><span style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px">_weakrefset.py the functions </span><span style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px">add(), </span><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px">remove(), discard(), __isub__(), etc are also using the problematic "ref()" function. The error could eventually happen there or even outside </span></font><span style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px">_weakrefset.py</span><span style="font-size:13.3333330154419px;color:rgb(0,0,0);font-family:Arial">.</span></div><div style><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px"><br></span></font></div><div style><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px">Looking forward some IronPython Guru can check the c# code. I will appreciate the help on that! In the meantime, I will use the __contains__() as above, trying to tackle the most common case.</span></font></div><div style><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px"><br></span></font></div><div style><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px">Regards,</span></font></div><div style><font color="#000000" face="Arial"><span style="font-size:13.3333330154419px">Andres Sommerhoff</span></font></div><div style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px"><br></div><div style="color:rgb(0,0,0);font-family:Arial;font-size:13.3333330154419px"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 13, 2015 at 11:55 AM, Andrew Graham <span dir="ltr"><<a href="mailto:andy@agraham.demon.co.uk" target="_blank">andy@agraham.demon.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div style="FONT-SIZE:10pt;FONT-FAMILY:'Arial';COLOR:#000000">
<div>I confirm the same problem, except that it happens more often for me, about
50 to 60 times each time I run the test. </div>
<div>It smells like some sort of “unsynchronised acess by different threads to
an object problem ”. </div>
<div> </div>
<div>One (very) hacky way round it until the real cause is established is to
modify __contains__() in _weakrefset.py line 68 to retry until it gets a proper
result</div>
<div> </div>
<div> def __contains__(self, item):</div>
<div> while 1:</div>
<div>
try:</div>
<div>
try:</div>
<div>
wr = ref(item)</div>
<div>
except TypeError:</div>
<div>
return False</div>
<div>
return wr in self.data</div>
<div> except
Exception:</div>
<div>
pass</div>
<div> </div>
<div>Regards</div>
<div> </div>
<div>Andy Graham</div>
<div style="FONT-SIZE:small;TEXT-DECORATION:none;FONT-FAMILY:"Calibri";FONT-WEIGHT:normal;COLOR:#000000;FONT-STYLE:normal;DISPLAY:inline">
<div style="FONT:10pt tahoma">
<div> </div>
<div style="BACKGROUND:#f5f5f5">
<div><b>From:</b> <a title="sommerhoff@gmail.com" href="mailto:sommerhoff@gmail.com" target="_blank">Andres Sommerhoff</a> </div>
<div><b>Sent:</b> Monday, April 13, 2015 6:39 AM</div>
<div><b>To:</b> <a title="ironpython-users@python.org" href="mailto:ironpython-users@python.org" target="_blank">ironpython-users@python.org</a> </div>
<div><b>Subject:</b> [Ironpython-users] Fwd: weakref random "SystemError" and
"ValueError" exception (affecting _weakrefset.py and abc.py)</div></div></div>
<div> </div></div>
<div style="FONT-SIZE:small;TEXT-DECORATION:none;FONT-FAMILY:"Calibri";FONT-WEIGHT:normal;COLOR:#000000;FONT-STYLE:normal;DISPLAY:inline"><div><div class="h5">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">
<div>Dear IronPython gurus!</div>
<div><font size="2" face="Arial"></font> </div>
<div>Hopping you can help me with a kind of ramdom bug (unexpected SystemError
and ValueError exception) in _weakrefset.py (it would be great if you can
replicate it by running my script below. Please let me know).
<div> </div>
<div>The error is random, but I managed to reproduce it by running the
"ramdomly" ofending code 100.000 times inside a loop (Script attached). Note
that the test takes only a few seconds, and in my PC it throws regularly between
5 to 30 exception for all those cycles). I see there are other people suffering
for the same, but without solution or workaround yet (<a href="https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html" target="_blank">https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html</a>
and <a href="https://github.com/IronLanguages/main/issues/1187" target="_blank">https://github.com/IronLanguages/main/issues/1187</a>)</div></div>
<p>In my case, weakref error was related with an intensive use of isinstance()
inside "Pulp" library (an optimization python library). <u>Just for your
reference</u>:<em> isintance() use internally a WeakSet as a cache for the class
types, which in turn use weakref (see ABCMeta.__instancecheck__() in the
standard file "abc.py"). </em></p>
<div>In my test script I have isolated the problem to WeakSet only (I isolated
it to clean the bug from the "abc.py" and "Pulp" library stuff). The exception
happens inside <strong>WeakSet.__contains__()</strong> function (from
_weakrefset.py file). As stated, it happens randomly, but apparently only
related with a GC collect cycle into a memory "hungry" python script . I ran the
test script in two PCs with similar results: Windows 7 64bits and Windows 7
32bits. Both using ipy.exe 32bit version 2.7.4 (2.7.0.40). The .NET version is
4.0.30319.18444 (32-bit). The test script does: </div>
<ol>
<li>It simulate a "memory intensive" python code (creating a weakref object of
2kb to 12kb in each loop. If smaller, like 0.1kb objects, then the bug don't
show up)
</li><li>It manually runs GC.collect() every 1.000 cycles (should collect those
weakref objects)
</li><li>... and it repeat (100.000 times) the following "offending" boolean
test:</li></ol>
<div><font face="monospace,monospace"> <font style="BACKGROUND-COLOR:rgb(234,153,153)">test = item in
WeakSetObject</font> <font color="#9900ff"></font><em><font color="#999999">#<- Repeated 100.000 times. </font></em></font></div>
<div><font color="#999999" face="monospace,monospace"><em>
#-> it fails between 10 to 20 times with an unexpected
exception</em></font></div>
<div><font face="Courier New"></font>
<div><font face="Courier New"></font> </div></div>
<div><font color="#0000ff"><strong>NOTE 1:</strong> The "item" is an object added
to the WeakSet at the beginning of the script, but "item" should not GC
collected as it also keeps a normal (not weak) reference alive. </font></div>
<div>
<div><font size="2" face="Arial"></font> </div></div>
<div><font color="#0000ff"><strong>NOTE 2:</strong> The boolean test should give
always True, which is the case 99.9% of the time. Less than 0.01%, the boolean
test fails raising an exception of the type "ValueError" <em>(Description:"Index
was out of range") </em>or a bit more frequent "SystemError"
<em>(Description:"Handle is not initialized")</em>. This happens 5 to 30 times
in 100.000 test cycle (Seems very small, but it is important enough to avoid a
practical construction of a medium size optimization problem with "Pulp"
library). </font></div>
<p>Tracking down the error, the exception ValueError is raised in line 70 and
the exception "SystemError" in line 73 of "_weakrefset.py" .</p>
<p> <font face="monospace,monospace">On
"Lib\_weakrefset.py"</font>
</p><div><font face="monospace,monospace"> 35 :class
WeakSet(object):</font></div>
<div><font face="monospace,monospace"> ....</font></div>
<div><font face="monospace,monospace"> 68 :
def __contains__(self, item):</font></div>
<div><font face="monospace,monospace"> 69
: try:</font></div>
<div><font face="monospace,monospace">
<strong>*70</strong>:
wr = <font color="#ff0000">ref</font>(item) <font color="#00ff00"># <- here is raised "ValueError" Exception ("Index was out of
range")</font></font></div>
<div><font face="monospace,monospace"> 71
: except TypeError:</font></div>
<div><font face="monospace,monospace"> 72
: return
False</font></div>
<div><font face="monospace,monospace">
<strong>*73</strong>: return <font color="#ff0000">wr</font> in self.data <font color="#00ff00"># <- here is raised
"SystemError" Exception ("Handle is not initialized")</font></font></div>
<p>Continuing after the exception, when executing the same boolean test again,
it works fine (same item, same WeakSetObject, same execution, local and global
context. Script was not started again!). I.e. if you catch the exception and
continue the execution, It is like as if the exception never happened before
(it's like a runtime lapsus!). </p>
<p>I believe to fix the source of the problem, Ironpython should trap or avoid
the error in weakref module itself (C# code). I don't know how... can someone
kindly help me to fix this?</p>
<div>Cheers,
<div>Andres Sommerhoff</div></div>
<div>
<div> </div></div>
<div>
<div> </div></div><p></p></div></div>
<div><font size="2" face="Arial"></font> </div></div>
</div></div><p>
</p><hr>
_______________________________________________<br>Ironpython-users mailing
list<br><a href="mailto:Ironpython-users@python.org" target="_blank">Ironpython-users@python.org</a><br><a href="https://mail.python.org/mailman/listinfo/ironpython-users" target="_blank">https://mail.python.org/mailman/listinfo/ironpython-users</a><br><p></p></div></div></div></div>
</blockquote></div><br></div>