[Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)

[Pawel Jasinski]

did anybody try it with the 2.7.4?
I wonder it is something we introduced in 2.7.5.
--pawel

On Mon, Apr 13, 2015 at 6:29 PM, Andres Sommerhoff <sommerhoff at gmail.com> wrote:
> Thank you Andrew, good to know you could replicate it. I went to very
> similar workarround with __contains__(), but yours is more elegant with the
> "while 1". However, I will recommend to repeat only a few times (to avoid a
> possible endless loop) and to catch the two unexpected exception only
> (SystemError, ValueError). So I suggest the following workaround based in
> your proposal should be:
>
>     def __contains__(self, item):
>         while 1:
>             try:
>                 try:
>                     wr = ref(item)
>                 except TypeError:
>                     return False
>                 return wr in self.data
>             except (SystemError, ValueError): #i.e: Don't catch
> KeyboardInterrupt, MemoryError, etc
>                 try: i+=1
>                 except: i=0       # i declared here. Don't want waste
> resource outside... (not Pythonic, I know)
>                 if i==10: raise   # Retry max 10 times and re-raise
>
> However, would like if someone can debug the error in the C# code of
> weakref. I'm afraid that the workaround will leave some potential error open
> beside __contains__(). I see than inside _weakrefset.py the functions add(),
> remove(), discard(), __isub__(), etc are also using the problematic "ref()"
> function. The error could eventually happen there or even outside
> _weakrefset.py.
>
> Looking forward some IronPython Guru can check the c# code. I will
> appreciate the help on that! In the meantime, I will use the __contains__()
> as above, trying to tackle the most common case.
>
> Regards,
> Andres Sommerhoff
>
>
>
> On Mon, Apr 13, 2015 at 11:55 AM, Andrew Graham <andy at agraham.demon.co.uk>
> wrote:
>>
>> I confirm the same problem, except that it happens more often for me,
>> about 50 to 60 times each time I run the test.
>> It smells like some sort of “unsynchronised acess by different threads to
>> an object problem ”.
>>
>> One (very) hacky way round it until the real cause is established is to
>> modify __contains__() in _weakrefset.py line 68 to retry until it gets a
>> proper result
>>
>>     def __contains__(self, item):
>>         while 1:
>>             try:
>>                 try:
>>                     wr = ref(item)
>>                 except TypeError:
>>                     return False
>>                 return wr in self.data
>>             except Exception:
>>                 pass
>>
>> Regards
>>
>> Andy Graham
>>
>> From: Andres Sommerhoff
>> Sent: Monday, April 13, 2015 6:39 AM
>> To: ironpython-users at python.org
>> Subject: [Ironpython-users] Fwd: weakref random "SystemError" and
>> "ValueError" exception (affecting _weakrefset.py and abc.py)
>>
>> Dear IronPython gurus!
>>
>> Hopping you can help me with a kind of ramdom bug (unexpected SystemError
>> and ValueError exception) in _weakrefset.py (it would be great if you can
>> replicate it by running my script below. Please let me know).
>>
>> The error is random, but I managed to reproduce it by running the
>> "ramdomly" ofending code 100.000 times inside a loop (Script attached). Note
>> that the test takes only a few seconds, and in my PC it throws regularly
>> between 5 to 30 exception for all those cycles). I see there are other
>> people suffering for the same, but without solution or workaround yet
>> (https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html
>> and https://github.com/IronLanguages/main/issues/1187)
>>
>> In my case, weakref error was related with an intensive use of
>> isinstance() inside "Pulp" library (an optimization python library). Just
>> for your reference: isintance() use internally a WeakSet as a cache for the
>> class types, which in turn use weakref (see ABCMeta.__instancecheck__() in
>> the standard file "abc.py").
>>
>> In my test script I have isolated the problem to WeakSet only (I isolated
>> it to clean the bug from the "abc.py" and "Pulp" library stuff). The
>> exception happens inside WeakSet.__contains__() function (from
>> _weakrefset.py file). As stated, it happens randomly, but apparently only
>> related with a GC collect cycle into a memory "hungry" python script . I ran
>> the test script in two PCs with similar results: Windows 7 64bits and
>> Windows 7 32bits. Both using ipy.exe 32bit version 2.7.4 (2.7.0.40). The
>> .NET version is 4.0.30319.18444 (32-bit). The test script does:
>>
>> It simulate a "memory intensive" python code (creating a weakref object of
>> 2kb to 12kb in each loop. If smaller, like 0.1kb objects, then the bug don't
>> show up)
>> It manually runs GC.collect() every 1.000 cycles (should collect those
>> weakref objects)
>> ... and it repeat (100.000 times) the following "offending" boolean test:
>>
>>        test = item in WeakSetObject  #<- Repeated 100.000 times.
>>                                       #-> it fails between 10 to 20 times
>> with an unexpected exception
>>
>> NOTE 1: The "item" is an object added to the WeakSet at the beginning of
>> the script, but "item" should not GC collected as it also keeps a normal
>> (not weak) reference alive.
>>
>> NOTE 2: The boolean test should give always True, which is the case 99.9%
>> of the time. Less than 0.01%, the boolean test fails raising an exception of
>> the type "ValueError" (Description:"Index was out of range") or a bit more
>> frequent "SystemError" (Description:"Handle is not initialized"). This
>> happens 5 to 30 times in 100.000 test cycle (Seems very small, but it is
>> important enough to avoid a practical construction of a medium size
>> optimization problem with "Pulp" library).
>>
>> Tracking down the error, the exception ValueError is raised in line 70 and
>> the exception "SystemError" in line 73 of "_weakrefset.py" .
>>
>>     On "Lib\_weakrefset.py"
>>
>>     35 :class WeakSet(object):
>>     ....
>>     68 :    def __contains__(self, item):
>>     69 :        try:
>>     *70:           wr = ref(item)     # <- here is raised "ValueError"
>> Exception ("Index was out of range")
>>     71 :        except TypeError:
>>     72 :            return False
>>     *73:       return wr in self.data # <- here is raised "SystemError"
>> Exception ("Handle is not initialized")
>>
>> Continuing after the exception, when executing the same boolean test
>> again, it works fine (same item, same WeakSetObject, same execution, local
>> and global context. Script was not started again!). I.e. if you catch the
>> exception and continue the execution, It is like as if the exception never
>> happened before (it's like a runtime lapsus!).
>>
>> I believe to fix the source of the problem, Ironpython should trap or
>> avoid the error in weakref module itself (C# code). I don't know how... can
>> someone kindly help me to fix this?
>>
>> Cheers,
>> Andres Sommerhoff
>>
>>
>>
>>
>>
>> ________________________________
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>
>
>
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>