[Ironpython-users] Fwd: weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)
Andres Sommerhoff
sommerhoff at gmail.com
Tue Apr 14 02:13:23 CEST 2015
Hi Pawel, its actually very old to me (more than a year). I'm suffering gradually as the code went more complex. At the beginning was not a big issue. Now, it is not a chance to run some code under IronPython without the weakref exception (so Im running it in Cpython or with the workarround of my previous email). Please note that I have tested only with 2.7.4, not with 2.7.5 yet (as 2.7.4 is the version that comes with SolverStudio software)
Can you try the script yourself?
Thank you for helping on this!
Regards,
Andres
> El 13-04-2015, a las 19:41, Pawel Jasinski <pawel.jasinski at gmail.com> escribió:
>
> did anybody try it with the 2.7.4?
> I wonder it is something we introduced in 2.7.5.
> --pawel
>
>> On Mon, Apr 13, 2015 at 6:29 PM, Andres Sommerhoff <sommerhoff at gmail.com> wrote:
>> Thank you Andrew, good to know you could replicate it. I went to very
>> similar workarround with __contains__(), but yours is more elegant with the
>> "while 1". However, I will recommend to repeat only a few times (to avoid a
>> possible endless loop) and to catch the two unexpected exception only
>> (SystemError, ValueError). So I suggest the following workaround based in
>> your proposal should be:
>>
>> def __contains__(self, item):
>> while 1:
>> try:
>> try:
>> wr = ref(item)
>> except TypeError:
>> return False
>> return wr in self.data
>> except (SystemError, ValueError): #i.e: Don't catch
>> KeyboardInterrupt, MemoryError, etc
>> try: i+=1
>> except: i=0 # i declared here. Don't want waste
>> resource outside... (not Pythonic, I know)
>> if i==10: raise # Retry max 10 times and re-raise
>>
>> However, would like if someone can debug the error in the C# code of
>> weakref. I'm afraid that the workaround will leave some potential error open
>> beside __contains__(). I see than inside _weakrefset.py the functions add(),
>> remove(), discard(), __isub__(), etc are also using the problematic "ref()"
>> function. The error could eventually happen there or even outside
>> _weakrefset.py.
>>
>> Looking forward some IronPython Guru can check the c# code. I will
>> appreciate the help on that! In the meantime, I will use the __contains__()
>> as above, trying to tackle the most common case.
>>
>> Regards,
>> Andres Sommerhoff
>>
>>
>>
>> On Mon, Apr 13, 2015 at 11:55 AM, Andrew Graham <andy at agraham.demon.co.uk>
>> wrote:
>>>
>>> I confirm the same problem, except that it happens more often for me,
>>> about 50 to 60 times each time I run the test.
>>> It smells like some sort of “unsynchronised acess by different threads to
>>> an object problem ”.
>>>
>>> One (very) hacky way round it until the real cause is established is to
>>> modify __contains__() in _weakrefset.py line 68 to retry until it gets a
>>> proper result
>>>
>>> def __contains__(self, item):
>>> while 1:
>>> try:
>>> try:
>>> wr = ref(item)
>>> except TypeError:
>>> return False
>>> return wr in self.data
>>> except Exception:
>>> pass
>>>
>>> Regards
>>>
>>> Andy Graham
>>>
>>> From: Andres Sommerhoff
>>> Sent: Monday, April 13, 2015 6:39 AM
>>> To: ironpython-users at python.org
>>> Subject: [Ironpython-users] Fwd: weakref random "SystemError" and
>>> "ValueError" exception (affecting _weakrefset.py and abc.py)
>>>
>>> Dear IronPython gurus!
>>>
>>> Hopping you can help me with a kind of ramdom bug (unexpected SystemError
>>> and ValueError exception) in _weakrefset.py (it would be great if you can
>>> replicate it by running my script below. Please let me know).
>>>
>>> The error is random, but I managed to reproduce it by running the
>>> "ramdomly" ofending code 100.000 times inside a loop (Script attached). Note
>>> that the test takes only a few seconds, and in my PC it throws regularly
>>> between 5 to 30 exception for all those cycles). I see there are other
>>> people suffering for the same, but without solution or workaround yet
>>> (https://mail.python.org/pipermail/ironpython-users/2014-November/017332.html
>>> and https://github.com/IronLanguages/main/issues/1187)
>>>
>>> In my case, weakref error was related with an intensive use of
>>> isinstance() inside "Pulp" library (an optimization python library). Just
>>> for your reference: isintance() use internally a WeakSet as a cache for the
>>> class types, which in turn use weakref (see ABCMeta.__instancecheck__() in
>>> the standard file "abc.py").
>>>
>>> In my test script I have isolated the problem to WeakSet only (I isolated
>>> it to clean the bug from the "abc.py" and "Pulp" library stuff). The
>>> exception happens inside WeakSet.__contains__() function (from
>>> _weakrefset.py file). As stated, it happens randomly, but apparently only
>>> related with a GC collect cycle into a memory "hungry" python script . I ran
>>> the test script in two PCs with similar results: Windows 7 64bits and
>>> Windows 7 32bits. Both using ipy.exe 32bit version 2.7.4 (2.7.0.40). The
>>> .NET version is 4.0.30319.18444 (32-bit). The test script does:
>>>
>>> It simulate a "memory intensive" python code (creating a weakref object of
>>> 2kb to 12kb in each loop. If smaller, like 0.1kb objects, then the bug don't
>>> show up)
>>> It manually runs GC.collect() every 1.000 cycles (should collect those
>>> weakref objects)
>>> ... and it repeat (100.000 times) the following "offending" boolean test:
>>>
>>> test = item in WeakSetObject #<- Repeated 100.000 times.
>>> #-> it fails between 10 to 20 times
>>> with an unexpected exception
>>>
>>> NOTE 1: The "item" is an object added to the WeakSet at the beginning of
>>> the script, but "item" should not GC collected as it also keeps a normal
>>> (not weak) reference alive.
>>>
>>> NOTE 2: The boolean test should give always True, which is the case 99.9%
>>> of the time. Less than 0.01%, the boolean test fails raising an exception of
>>> the type "ValueError" (Description:"Index was out of range") or a bit more
>>> frequent "SystemError" (Description:"Handle is not initialized"). This
>>> happens 5 to 30 times in 100.000 test cycle (Seems very small, but it is
>>> important enough to avoid a practical construction of a medium size
>>> optimization problem with "Pulp" library).
>>>
>>> Tracking down the error, the exception ValueError is raised in line 70 and
>>> the exception "SystemError" in line 73 of "_weakrefset.py" .
>>>
>>> On "Lib\_weakrefset.py"
>>>
>>> 35 :class WeakSet(object):
>>> ....
>>> 68 : def __contains__(self, item):
>>> 69 : try:
>>> *70: wr = ref(item) # <- here is raised "ValueError"
>>> Exception ("Index was out of range")
>>> 71 : except TypeError:
>>> 72 : return False
>>> *73: return wr in self.data # <- here is raised "SystemError"
>>> Exception ("Handle is not initialized")
>>>
>>> Continuing after the exception, when executing the same boolean test
>>> again, it works fine (same item, same WeakSetObject, same execution, local
>>> and global context. Script was not started again!). I.e. if you catch the
>>> exception and continue the execution, It is like as if the exception never
>>> happened before (it's like a runtime lapsus!).
>>>
>>> I believe to fix the source of the problem, Ironpython should trap or
>>> avoid the error in weakref module itself (C# code). I don't know how... can
>>> someone kindly help me to fix this?
>>>
>>> Cheers,
>>> Andres Sommerhoff
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>> _______________________________________________
>>> Ironpython-users mailing list
>>> Ironpython-users at python.org
>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>
>>
>>
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>>
More information about the Ironpython-users
mailing list