[Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)
Andres Sommerhoff
sommerhoff at gmail.com
Mon Apr 13 18:29:47 CEST 2015
Thank you Andrew, good to know you could replicate it. I went to very
similar workarround with __contains__(), but yours is more elegant with the
"while 1". However, I will recommend to repeat only a few times (to avoid a
possible endless loop) and to catch the two unexpected exception only
(SystemError,
ValueError). So I suggest the following workaround based in your proposal
should be:
def __contains__(self, item):
while 1:
try:
try:
wr = ref(item)
except TypeError:
return False
return wr in self.data
except (SystemError, ValueError): #i.e: Don't
catch KeyboardInterrupt, MemoryError, etc
try: i+=1
except: i=0 # i declared here. Don't want waste
resource outside... (not Pythonic, I know)
if i==10: raise # Retry max 10 times and re-raise
However, would like if someone can debug the error in the C# code of
weakref. I'm afraid that the workaround will leave some potential error
open beside __contains__(). I see than inside _weakrefset.py the functions
add(), remove(), discard(), __isub__(), etc are also using the problematic
"ref()" function. The error could eventually happen there or even outside
_weakrefset.py.
Looking forward some IronPython Guru can check the c# code. I
will appreciate the help on that! In the meantime, I will use the
__contains__() as above, trying to tackle the most common case.
Regards,
Andres Sommerhoff
On Mon, Apr 13, 2015 at 11:55 AM, Andrew Graham <andy at agraham.demon.co.uk>
wrote:
> I confirm the same problem, except that it happens more often for me,
> about 50 to 60 times each time I run the test.
> It smells like some sort of “unsynchronised acess by different threads to
> an object problem ”.
>
> One (very) hacky way round it until the real cause is established is to
> modify __contains__() in _weakrefset.py line 68 to retry until it gets a
> proper result
>
> def __contains__(self, item):
> while 1:
> try:
> try:
> wr = ref(item)
> except TypeError:
> return False
> return wr in self.data
> except Exception:
> pass
>
> Regards
>
> Andy Graham
>
> *From:* Andres Sommerhoff <sommerhoff at gmail.com>
> *Sent:* Monday, April 13, 2015 6:39 AM
> *To:* ironpython-users at python.org
> *Subject:* [Ironpython-users] Fwd: weakref random "SystemError" and
> "ValueError" exception (affecting _weakrefset.py and abc.py)
>
> Dear IronPython gurus!
>
> Hopping you can help me with a kind of ramdom bug (unexpected SystemError
> and ValueError exception) in _weakrefset.py (it would be great if you can
> replicate it by running my script below. Please let me know).
>
> The error is random, but I managed to reproduce it by running the
> "ramdomly" ofending code 100.000 times inside a loop (Script attached).
> Note that the test takes only a few seconds, and in my PC it throws
> regularly between 5 to 30 exception for all those cycles). I see there are
> other people suffering for the same, but without solution or workaround yet
> (
> https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html
> and https://github.com/IronLanguages/main/issues/1187)
>
> In my case, weakref error was related with an intensive use of
> isinstance() inside "Pulp" library (an optimization python library). *Just
> for your reference*:* isintance() use internally a WeakSet as a cache for
> the class types, which in turn use weakref (see ABCMeta.__instancecheck__()
> in the standard file "abc.py"). *
> In my test script I have isolated the problem to WeakSet only (I isolated
> it to clean the bug from the "abc.py" and "Pulp" library stuff). The
> exception happens inside *WeakSet.__contains__()* function (from
> _weakrefset.py file). As stated, it happens randomly, but apparently only
> related with a GC collect cycle into a memory "hungry" python script . I
> ran the test script in two PCs with similar results: Windows 7 64bits and
> Windows 7 32bits. Both using ipy.exe 32bit version 2.7.4 (2.7.0.40). The
> .NET version is 4.0.30319.18444 (32-bit). The test script does:
>
> 1. It simulate a "memory intensive" python code (creating a weakref
> object of 2kb to 12kb in each loop. If smaller, like 0.1kb objects, then
> the bug don't show up)
> 2. It manually runs GC.collect() every 1.000 cycles (should collect
> those weakref objects)
> 3. ... and it repeat (100.000 times) the following "offending" boolean
> test:
>
> test = item in WeakSetObject *#<- Repeated 100.000 times. *
> * #-> it fails between 10 to 20 times
> with an unexpected exception*
>
> *NOTE 1:* The "item" is an object added to the WeakSet at the beginning
> of the script, but "item" should not GC collected as it also keeps a normal
> (not weak) reference alive.
>
> *NOTE 2:* The boolean test should give always True, which is the case
> 99.9% of the time. Less than 0.01%, the boolean test fails raising an
> exception of the type "ValueError" *(Description:"Index was out of
> range") *or a bit more frequent "SystemError" *(Description:"Handle is
> not initialized")*. This happens 5 to 30 times in 100.000 test cycle
> (Seems very small, but it is important enough to avoid a practical
> construction of a medium size optimization problem with "Pulp" library).
>
> Tracking down the error, the exception ValueError is raised in line 70 and
> the exception "SystemError" in line 73 of "_weakrefset.py" .
>
> On "Lib\_weakrefset.py"
> 35 :class WeakSet(object):
> ....
> 68 : def __contains__(self, item):
> 69 : try:
> **70*: wr = ref(item) # <- here is raised "ValueError"
> Exception ("Index was out of range")
> 71 : except TypeError:
> 72 : return False
> **73*: return wr in self.data # <- here is raised "SystemError"
> Exception ("Handle is not initialized")
>
> Continuing after the exception, when executing the same boolean test
> again, it works fine (same item, same WeakSetObject, same execution, local
> and global context. Script was not started again!). I.e. if you catch the
> exception and continue the execution, It is like as if the exception never
> happened before (it's like a runtime lapsus!).
>
> I believe to fix the source of the problem, Ironpython should trap or
> avoid the error in weakref module itself (C# code). I don't know how... can
> someone kindly help me to fix this?
> Cheers,
> Andres Sommerhoff
>
>
>
>
>
> ------------------------------
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20150413/1cb43f8f/attachment.html>
More information about the Ironpython-users
mailing list