[Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)

Andres Sommerhoff sommerhoff at gmail.com
Mon Apr 13 18:29:47 CEST 2015

Thank you Andrew, good to know you could replicate it. I went to very
similar workarround with __contains__(), but yours is more elegant with the
"while 1". However, I will recommend to repeat only a few times (to avoid a
possible endless loop) and to catch the two unexpected exception only
ValueError). So I suggest the following workaround based in your proposal
should be:

    def __contains__(self, item):
        while 1:
                    wr = ref(item)
                except TypeError:
                    return False
                return wr in self.data
            except (SystemError, ValueError): #i.e: Don't
catch KeyboardInterrupt, MemoryError, etc
                try: i+=1
                except: i=0       # i declared here. Don't want waste
resource outside... (not Pythonic, I know)
                if i==10: raise   # Retry max 10 times and re-raise

However, would like if someone can debug the error in the C# code of
weakref. I'm afraid that the workaround will leave some potential error
open beside __contains__(). I see than inside _weakrefset.py the functions
add(), remove(), discard(), __isub__(), etc are also using the problematic
"ref()" function. The error could eventually happen there or even outside

Looking forward some IronPython Guru can check the c# code. I
will appreciate the help on that! In the meantime, I will use the
__contains__() as above, trying to tackle the most common case.

Andres Sommerhoff

On Mon, Apr 13, 2015 at 11:55 AM, Andrew Graham <andy at agraham.demon.co.uk>

>   I confirm the same problem, except that it happens more often for me,
> about 50 to 60 times each time I run the test.
> It smells like some sort of “unsynchronised acess by different threads to
> an object problem ”.
> One (very) hacky way round it until the real cause is established is to
> modify __contains__() in _weakrefset.py line 68 to retry until it gets a
> proper result
>     def __contains__(self, item):
>         while 1:
>             try:
>                 try:
>                     wr = ref(item)
>                 except TypeError:
>                     return False
>                 return wr in self.data
>             except Exception:
>                 pass
> Regards
> Andy Graham
>  *From:* Andres Sommerhoff <sommerhoff at gmail.com>
> *Sent:* Monday, April 13, 2015 6:39 AM
> *To:* ironpython-users at python.org
> *Subject:* [Ironpython-users] Fwd: weakref random "SystemError" and
> "ValueError" exception (affecting _weakrefset.py and abc.py)
>   Dear IronPython gurus!
> Hopping you can help me with a kind of ramdom bug (unexpected SystemError
> and ValueError exception) in _weakrefset.py (it would be great if you can
> replicate it by running my script below. Please let me know).
> The error is random, but I managed to reproduce it by running the
> "ramdomly" ofending code 100.000 times inside a loop (Script attached).
> Note that the test takes only a few seconds, and in my PC it throws
> regularly between 5 to 30 exception for all those cycles). I see there are
> other people suffering for the same, but without solution or workaround yet
> (
> https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html
> and https://github.com/IronLanguages/main/issues/1187)
> In my case, weakref error was related with an intensive use of
> isinstance() inside "Pulp" library (an optimization python library). *Just
> for your reference*:* isintance() use internally a WeakSet as a cache for
> the class types, which in turn use weakref (see ABCMeta.__instancecheck__()
> in the standard file "abc.py"). *
> In my test script I have isolated the problem to WeakSet only (I isolated
> it to clean the bug from the "abc.py" and "Pulp" library stuff). The
> exception happens inside *WeakSet.__contains__()* function (from
> _weakrefset.py file). As stated, it happens randomly, but apparently only
> related with a GC collect cycle into a memory "hungry" python script . I
> ran the test script in two PCs with similar results: Windows 7 64bits and
> Windows 7 32bits. Both using ipy.exe 32bit version 2.7.4 ( The
> .NET version is 4.0.30319.18444 (32-bit). The test script does:
>    1. It simulate a "memory intensive" python code (creating a weakref
>    object of 2kb to 12kb in each loop. If smaller, like 0.1kb objects, then
>    the bug don't show up)
>    2. It manually runs GC.collect() every 1.000 cycles (should collect
>    those weakref objects)
>    3. ... and it repeat (100.000 times) the following "offending" boolean
>    test:
>        test = item in WeakSetObject  *#<- Repeated 100.000 times. *
> *                                      #-> it fails between 10 to 20 times
> with an unexpected exception*
> *NOTE 1:* The "item" is an object added to the WeakSet at the beginning
> of the script, but "item" should not GC collected as it also keeps a normal
> (not weak) reference alive.
> *NOTE 2:* The boolean test should give always True, which is the case
> 99.9% of the time. Less than 0.01%, the boolean test fails raising an
> exception of the type "ValueError" *(Description:"Index was out of
> range") *or a bit more frequent "SystemError" *(Description:"Handle is
> not initialized")*. This happens 5 to 30 times in 100.000 test cycle
> (Seems very small, but it is important enough to avoid a practical
> construction of a medium size optimization problem with "Pulp" library).
> Tracking down the error, the exception ValueError is raised in line 70 and
> the exception "SystemError" in line 73 of "_weakrefset.py" .
>     On "Lib\_weakrefset.py"
>     35 :class WeakSet(object):
>     ....
>     68 :    def __contains__(self, item):
>     69 :        try:
>     **70*:           wr = ref(item)     # <- here is raised "ValueError"
> Exception ("Index was out of range")
>     71 :        except TypeError:
>     72 :            return False
>     **73*:       return wr in self.data # <- here is raised "SystemError"
> Exception ("Handle is not initialized")
> Continuing after the exception, when executing the same boolean test
> again, it works fine (same item, same WeakSetObject, same execution, local
> and global context. Script was not started again!). I.e. if you catch the
> exception and continue the execution, It is like as if the exception never
> happened before (it's like a runtime lapsus!).
> I believe to fix the source of the problem, Ironpython should trap or
> avoid the error in weakref module itself (C# code). I don't know how... can
> someone kindly help me to fix this?
> Cheers,
> Andres Sommerhoff
> ------------------------------
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20150413/1cb43f8f/attachment.html>

More information about the Ironpython-users mailing list