[Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)

Pawel Jasinski pawel.jasinski at gmail.com
Tue Apr 14 11:15:08 CEST 2015 

I can reproduce it with 2.7.4 and 2.7.5.
--pawel

On Tue, Apr 14, 2015 at 2:13 AM, Andres Sommerhoff <sommerhoff at gmail.com> wrote:
> Hi Pawel, its actually very old to me (more than a year). I'm suffering gradually as the code went more complex. At the beginning was not a big issue. Now, it is not a chance to run some code under IronPython without the weakref exception (so Im running it in Cpython or with the workarround of my previous email). Please note that I have tested only with 2.7.4, not with 2.7.5 yet (as 2.7.4 is the version that comes with SolverStudio software)
>
> Can you try the script yourself?
>
> Thank you for helping on this!
>
> Regards,
> Andres
>
>> El 13-04-2015, a las 19:41, Pawel Jasinski <pawel.jasinski at gmail.com> escribió:
>>
>> did anybody try it with the 2.7.4?
>> I wonder it is something we introduced in 2.7.5.
>> --pawel
>>
>>> On Mon, Apr 13, 2015 at 6:29 PM, Andres Sommerhoff <sommerhoff at gmail.com> wrote:
>>> Thank you Andrew, good to know you could replicate it. I went to very
>>> similar workarround with __contains__(), but yours is more elegant with the
>>> "while 1". However, I will recommend to repeat only a few times (to avoid a
>>> possible endless loop) and to catch the two unexpected exception only
>>> (SystemError, ValueError). So I suggest the following workaround based in
>>> your proposal should be:
>>>
>>>    def __contains__(self, item):
>>>        while 1:
>>>            try:
>>>                try:
>>>                    wr = ref(item)
>>>                except TypeError:
>>>                    return False
>>>                return wr in self.data
>>>            except (SystemError, ValueError): #i.e: Don't catch
>>> KeyboardInterrupt, MemoryError, etc
>>>                try: i+=1
>>>                except: i=0       # i declared here. Don't want waste
>>> resource outside... (not Pythonic, I know)
>>>                if i==10: raise   # Retry max 10 times and re-raise
>>>
>>> However, would like if someone can debug the error in the C# code of
>>> weakref. I'm afraid that the workaround will leave some potential error open
>>> beside __contains__(). I see than inside _weakrefset.py the functions add(),
>>> remove(), discard(), __isub__(), etc are also using the problematic "ref()"
>>> function. The error could eventually happen there or even outside
>>> _weakrefset.py.
>>>
>>> Looking forward some IronPython Guru can check the c# code. I will
>>> appreciate the help on that! In the meantime, I will use the __contains__()
>>> as above, trying to tackle the most common case.
>>>
>>> Regards,
>>> Andres Sommerhoff
>>>
>>>
>>>
>>> On Mon, Apr 13, 2015 at 11:55 AM, Andrew Graham <andy at agraham.demon.co.uk>
>>> wrote:
>>>>
>>>> I confirm the same problem, except that it happens more often for me,
>>>> about 50 to 60 times each time I run the test.
>>>> It smells like some sort of “unsynchronised acess by different threads to
>>>> an object problem ”.
>>>>
>>>> One (very) hacky way round it until the real cause is established is to
>>>> modify __contains__() in _weakrefset.py line 68 to retry until it gets a
>>>> proper result
>>>>
>>>>    def __contains__(self, item):
>>>>        while 1:
>>>>            try:
>>>>                try:
>>>>                    wr = ref(item)
>>>>                except TypeError:
>>>>                    return False
>>>>                return wr in self.data
>>>>            except Exception:
>>>>                pass
>>>>
>>>> Regards
>>>>
>>>> Andy Graham
>>>>
>>>> From: Andres Sommerhoff
>>>> Sent: Monday, April 13, 2015 6:39 AM
>>>> To: ironpython-users at python.org
>>>> Subject: [Ironpython-users] Fwd: weakref random "SystemError" and
>>>> "ValueError" exception (affecting _weakrefset.py and abc.py)
>>>>
>>>> Dear IronPython gurus!
>>>>
>>>> Hopping you can help me with a kind of ramdom bug (unexpected SystemError
>>>> and ValueError exception) in _weakrefset.py (it would be great if you can
>>>> replicate it by running my script below. Please let me know).
>>>>
>>>> The error is random, but I managed to reproduce it by running the
>>>> "ramdomly" ofending code 100.000 times inside a loop (Script attached). Note
>>>> that the test takes only a few seconds, and in my PC it throws regularly
>>>> between 5 to 30 exception for all those cycles). I see there are other
>>>> people suffering for the same, but without solution or workaround yet
>>>> (https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html
>>>> and https://github.com/IronLanguages/main/issues/1187)
>>>>
>>>> In my case, weakref error was related with an intensive use of
>>>> isinstance() inside "Pulp" library (an optimization python library). Just
>>>> for your reference: isintance() use internally a WeakSet as a cache for the
>>>> class types, which in turn use weakref (see ABCMeta.__instancecheck__() in
>>>> the standard file "abc.py").
>>>>
>>>> In my test script I have isolated the problem to WeakSet only (I isolated
>>>> it to clean the bug from the "abc.py" and "Pulp" library stuff). The
>>>> exception happens inside WeakSet.__contains__() function (from
>>>> _weakrefset.py file). As stated, it happens randomly, but apparently only
>>>> related with a GC collect cycle into a memory "hungry" python script . I ran
>>>> the test script in two PCs with similar results: Windows 7 64bits and
>>>> Windows 7 32bits. Both using ipy.exe 32bit version 2.7.4 (2.7.0.40). The
>>>> .NET version is 4.0.30319.18444 (32-bit). The test script does:
>>>>
>>>> It simulate a "memory intensive" python code (creating a weakref object of
>>>> 2kb to 12kb in each loop. If smaller, like 0.1kb objects, then the bug don't
>>>> show up)
>>>> It manually runs GC.collect() every 1.000 cycles (should collect those
>>>> weakref objects)
>>>> ... and it repeat (100.000 times) the following "offending" boolean test:
>>>>
>>>>       test = item in WeakSetObject  #<- Repeated 100.000 times.
>>>>                                      #-> it fails between 10 to 20 times
>>>> with an unexpected exception
>>>>
>>>> NOTE 1: The "item" is an object added to the WeakSet at the beginning of
>>>> the script, but "item" should not GC collected as it also keeps a normal
>>>> (not weak) reference alive.
>>>>
>>>> NOTE 2: The boolean test should give always True, which is the case 99.9%
>>>> of the time. Less than 0.01%, the boolean test fails raising an exception of
>>>> the type "ValueError" (Description:"Index was out of range") or a bit more
>>>> frequent "SystemError" (Description:"Handle is not initialized"). This
>>>> happens 5 to 30 times in 100.000 test cycle (Seems very small, but it is
>>>> important enough to avoid a practical construction of a medium size
>>>> optimization problem with "Pulp" library).
>>>>
>>>> Tracking down the error, the exception ValueError is raised in line 70 and
>>>> the exception "SystemError" in line 73 of "_weakrefset.py" .
>>>>
>>>>    On "Lib\_weakrefset.py"
>>>>
>>>>    35 :class WeakSet(object):
>>>>    ....
>>>>    68 :    def __contains__(self, item):
>>>>    69 :        try:
>>>>    *70:           wr = ref(item)     # <- here is raised "ValueError"
>>>> Exception ("Index was out of range")
>>>>    71 :        except TypeError:
>>>>    72 :            return False
>>>>    *73:       return wr in self.data # <- here is raised "SystemError"
>>>> Exception ("Handle is not initialized")
>>>>
>>>> Continuing after the exception, when executing the same boolean test
>>>> again, it works fine (same item, same WeakSetObject, same execution, local
>>>> and global context. Script was not started again!). I.e. if you catch the
>>>> exception and continue the execution, It is like as if the exception never
>>>> happened before (it's like a runtime lapsus!).
>>>>
>>>> I believe to fix the source of the problem, Ironpython should trap or
>>>> avoid the error in weakref module itself (C# code). I don't know how... can
>>>> someone kindly help me to fix this?
>>>>
>>>> Cheers,
>>>> Andres Sommerhoff
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> _______________________________________________
>>>> Ironpython-users mailing list
>>>> Ironpython-users at python.org
>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>
>>>
>>>
>>> _______________________________________________
>>> Ironpython-users mailing list
>>> Ironpython-users at python.org
>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>

More information about the Ironpython-users mailing list