Oops, forgot to cc pypy-dev... Carl Friedrich -------- Original Message -------- From: Carl Friedrich Bolz <cfbolz@gmx.de> Sent: December 19, 2016 5:59:01 PM GMT+01:00 To: Frank Wang <frankw@mit.edu> Subject: Re: [pypy-dev] Adding init/variables to W_Root Hi Frank, The solution is to add an _attrs_ = ('__weakref__', 'rb_flags',) declaration to the body of W_Root. It's like __slots__, but only for RPython. Cheers, Carl Friedrich On December 19, 2016 5:42:12 PM GMT+01:00, Frank Wang <frankw@mit.edu> wrote:
Hi Carl, Thanks for the tip! It's mostly working. I get this error: [translation:ERROR] ImmutableConflictError: class <InstanceRepr for pypy.module._cffi_backend.libraryobj.W_Library> has _immutable_=True, but parent class <InstanceRepr for pypy.interpreter.baseobjspace.W_Root> defines (at least) the mutable field 'rb_flags' I'm tempted to set the flag to be False. Is there any reason that W_Root or any of its subclasses cannot contain immutable variables? Frank On Mon, Dec 19, 2016 at 9:00 AM, Carl Friedrich Bolz <cfbolz@gmx.de> wrote:
This also seems to happen in a lot of classes because W_Root is the parent class for many classes. Is there a way to universally turn off this immutable flag (assuming it's okay) in all classes? Frank On Mon, Dec 19, 2016 at 3:55 PM, Frank Wang <frankw@mit.edu> wrote:
Hi Frank, On 20 December 2016 at 01:29, Frank Wang <frankw@mit.edu> wrote:
You can hack at the place where the error is raised. Assuming you're translating without the JIT, it should not cause too much problems. As an alternative to adding an attribute to every object, you could also use a rpython.rlib.rweakref.RWeakKeyDictionary(W_Root, RbFlags), for some class RbFlags which has got the rb_flags dictionary and/or any other related data. The advantage is that it is a less intrusive change, reduces memory usage if many objects don't need the extra information at all, and avoids the immutable problem. The inconvenient is that going from the object to the rb_flags dictionary is slower. There are also alternatives that are better if you are ok with supporting not *all* objects. For example, you can edit objspace/std/mapdict.py to add a new dynamic attribute that is internally called ``"rbflags", SPECIAL``; see ``"weakref", SPECIAL`` for an existing example. This would work for all objects that use mapdicts, i.e. all instances of user-defined classes, and additionally instances of many built-in types as well---but not instances of the core types like "int" or "str" or "list" or "tuple" or "dict". A bientôt, Armin.
Hi Armin, I've taken your advice, and my W_Root has these additions (in bold): class W_Root(object): _attrs_ = ['__weakref__', *'rb_flags'*] _must_be_light_finalizer_ = True user_overridden_class = False # Riverbed flags * rb_flags = None* * def add_rbflags(self, new_flags):* * if self.rb_flags is None:* * self.rb_flags = {}* * for flag in new_flags:* * self.rb_flags[flag] = None* * def set_rbflags(self, new_flags):* * self.rb_flags = new_flags* * def get_rbflags(self):* * if self.rb_flags is None:* * self.rb_flags = {}* * return self.rb_flags* In pyopcode.py, I've made the following modifications. I've shown them in bold. The high level idea is that for a binary operation, I want to take the union of the rb_flags in w_1 and w_2 and set the union to be the rb_flags in w_result. def binaryoperation(operationname): """NOT_RPYTHON""" def opimpl(self, *ignored): operation = getattr(self.space, operationname) w_2 = self.popvalue() w_1 = self.popvalue() w_result = operation(w_1, w_2) # union flags in w_1 and w_2 and propagate to result * w_1_rbflags = w_1.get_rbflags()* * w_2_rbflags = w_2.get_rbflags()* * w_1_rbflags.update(w_2_rbflags)* * w_result.set_rbflags(w_1_rbflags)* self.pushvalue(w_result) opimpl.binop = operationname return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname) However, when I do this, I get the following error: [translation:ERROR] UnionError: Offending annotations: SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) SomeInstance(can_be_None=True, classdef=pypy.interpreter.baseobjspace.W_Root) v945 = setattr(self_268, ('rb_flags'), v944) In <FunctionGraph of (pypy.interpreter.baseobjspace:46)W_Root.get_rbflags at 0x40f26de8>: Happened at file /home/ubuntu/pypy2-v5.3.1-src/pypy/interpreter/baseobjspace.py line 48 ==> self.rb_flags = {} Known variable annotations: self_268 = SomeInstance(can_be_None=False, classdef=pypy.interpreter.baseobjspace.W_Root) v944 = SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) Processing block: block@15 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> in (pypy.interpreter.baseobjspace:46)W_Root.get_rbflags containing the following operations: v944 = newdict() v945 = setattr(self_268, ('rb_flags'), v944) --end-- Any thoughts on this? Frank On Mon, Dec 19, 2016 at 10:32 PM, Frank Wang <frankw@mit.edu> wrote:
On 12 January 2017 at 09:08, Frank Wang <frankw@mit.edu> wrote:
It looks like, if an app-level exception is in progress, w_result may be None. Try checking for that before setting the flags. Also: did you mean to alter w_1's rbflags? Seems strange that you'd alter w_1 but not w_2. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement.
Hi William, Good catch! I was just trying to get something to work, but it seems like you found another bug. I am now checking if it is None: # union flags in w_1 and w_2 and propagate to result if isinstance(w_result, W_Root) and w_result is not None: rb_results = {} if isinstance(w_1, W_Root) and w_1 is not None: w_1_rbflags = w_1.get_rbflags() rb_results.update(w_1_rbflags) if isinstance(w_2, W_Root) and w_2 is not None: w_2_rbflags = w_2.get_rbflags() rb_results.update(w_2_rbflags) w_result.set_rbflags(rb_results) I now get this error: [translation:ERROR] UnionError: Offending annotations: SomeInstance(can_be_None=False, classdef=pypy.interpreter.baseobjspace.W_Root) SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) Occurred processing the following simple_call: <MethodDesc 'set_rbflags' of <ClassDef 'pypy.interpreter.baseobjspace.W_Root'> bound to <ClassDef 'pypy.interpreter.baseobjspace.W_Root'> {}> returning v831 = simple_call(v830, rb_results_0) In <FunctionGraph of (pypy.interpreter.pyopcode:40)opcode_impl_for_div__star_2 at 0x1c190b10>: Happened at file /home/ubuntu/pypy2-v5.3.1-src/pypy/interpreter/pyopcode.py line 55 ==> w_result.set_rbflags(rb_results) Known variable annotations: v830 = SomePBC(can_be_None=False, descriptions={...1...}, knowntype=method, subset_of=None) rb_results_0 = SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) Processing block: block@203 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> in (pypy.interpreter.pyopcode:40)opcode_impl_for_div__star_2 containing the following operations: v830 = getattr(w_result_0, ('set_rbflags')) v831 = simple_call(v830, rb_results_0) --end-- I assume this happens because w_result can be a "subset" of None. What is the proper way to check if it's None? Frank On Wed, Jan 11, 2017 at 3:00 PM, William ML Leslie < william.leslie.ttg@gmail.com> wrote:
Hi, On 12 January 2017 at 03:35, Frank Wang <frankw@mit.edu> wrote:
Good catch! I was just trying to get something to work, but it seems like you found another bug. I am now checking if it is None:
No, I think these latest changes don't do anything. The error message says: you are storing in the rb_flags attribute something that is sometimes a dict, and sometimes an instance of W_Root. Check if there is somewhere else a subclass of W_Root with its own rb_flags attribute which would be set to a W_Root. Then you'd get a crash using that subclass, also untranslated, simply because 'rb_flags' is supposed to be two different things. To be on the safe side, you should change the name of the attribute in W_Root from rb_flags to something much more obscure like "_frank_rb_flags". A bientôt, Armin.
Re-hi, On 13 January 2017 at 12:34, Armin Rigo <armin.rigo@gmail.com> wrote:
says: you are storing in the rb_flags attribute something that is sometimes a dict, and sometimes an instance of W_Root.
Another potential reason would be that there is one buggy call somewhere that calls .set_rbflags(x) where x is an instance of W_Root instead of a dictionary. A bientôt, Armin.
Hi Armin, Thanks for the help! It seems like the problem is that somewhere in my code, I called .set_rbflags(x), and it's not checked that x is a dict. Frank On Fri, Jan 13, 2017 at 6:36 AM, Armin Rigo <armin.rigo@gmail.com> wrote:
Hi Carl, Thanks for the tip! It's mostly working. I get this error: [translation:ERROR] ImmutableConflictError: class <InstanceRepr for pypy.module._cffi_backend.libraryobj.W_Library> has _immutable_=True, but parent class <InstanceRepr for pypy.interpreter.baseobjspace.W_Root> defines (at least) the mutable field 'rb_flags' I'm tempted to set the flag to be False. Is there any reason that W_Root or any of its subclasses cannot contain immutable variables? Frank On Mon, Dec 19, 2016 at 9:00 AM, Carl Friedrich Bolz <cfbolz@gmx.de> wrote:
This also seems to happen in a lot of classes because W_Root is the parent class for many classes. Is there a way to universally turn off this immutable flag (assuming it's okay) in all classes? Frank On Mon, Dec 19, 2016 at 3:55 PM, Frank Wang <frankw@mit.edu> wrote:
Hi Frank, On 20 December 2016 at 01:29, Frank Wang <frankw@mit.edu> wrote:
You can hack at the place where the error is raised. Assuming you're translating without the JIT, it should not cause too much problems. As an alternative to adding an attribute to every object, you could also use a rpython.rlib.rweakref.RWeakKeyDictionary(W_Root, RbFlags), for some class RbFlags which has got the rb_flags dictionary and/or any other related data. The advantage is that it is a less intrusive change, reduces memory usage if many objects don't need the extra information at all, and avoids the immutable problem. The inconvenient is that going from the object to the rb_flags dictionary is slower. There are also alternatives that are better if you are ok with supporting not *all* objects. For example, you can edit objspace/std/mapdict.py to add a new dynamic attribute that is internally called ``"rbflags", SPECIAL``; see ``"weakref", SPECIAL`` for an existing example. This would work for all objects that use mapdicts, i.e. all instances of user-defined classes, and additionally instances of many built-in types as well---but not instances of the core types like "int" or "str" or "list" or "tuple" or "dict". A bientôt, Armin.
Hi Armin, I've taken your advice, and my W_Root has these additions (in bold): class W_Root(object): _attrs_ = ['__weakref__', *'rb_flags'*] _must_be_light_finalizer_ = True user_overridden_class = False # Riverbed flags * rb_flags = None* * def add_rbflags(self, new_flags):* * if self.rb_flags is None:* * self.rb_flags = {}* * for flag in new_flags:* * self.rb_flags[flag] = None* * def set_rbflags(self, new_flags):* * self.rb_flags = new_flags* * def get_rbflags(self):* * if self.rb_flags is None:* * self.rb_flags = {}* * return self.rb_flags* In pyopcode.py, I've made the following modifications. I've shown them in bold. The high level idea is that for a binary operation, I want to take the union of the rb_flags in w_1 and w_2 and set the union to be the rb_flags in w_result. def binaryoperation(operationname): """NOT_RPYTHON""" def opimpl(self, *ignored): operation = getattr(self.space, operationname) w_2 = self.popvalue() w_1 = self.popvalue() w_result = operation(w_1, w_2) # union flags in w_1 and w_2 and propagate to result * w_1_rbflags = w_1.get_rbflags()* * w_2_rbflags = w_2.get_rbflags()* * w_1_rbflags.update(w_2_rbflags)* * w_result.set_rbflags(w_1_rbflags)* self.pushvalue(w_result) opimpl.binop = operationname return func_with_new_name(opimpl, "opcode_impl_for_%s" % operationname) However, when I do this, I get the following error: [translation:ERROR] UnionError: Offending annotations: SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) SomeInstance(can_be_None=True, classdef=pypy.interpreter.baseobjspace.W_Root) v945 = setattr(self_268, ('rb_flags'), v944) In <FunctionGraph of (pypy.interpreter.baseobjspace:46)W_Root.get_rbflags at 0x40f26de8>: Happened at file /home/ubuntu/pypy2-v5.3.1-src/pypy/interpreter/baseobjspace.py line 48 ==> self.rb_flags = {} Known variable annotations: self_268 = SomeInstance(can_be_None=False, classdef=pypy.interpreter.baseobjspace.W_Root) v944 = SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) Processing block: block@15 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> in (pypy.interpreter.baseobjspace:46)W_Root.get_rbflags containing the following operations: v944 = newdict() v945 = setattr(self_268, ('rb_flags'), v944) --end-- Any thoughts on this? Frank On Mon, Dec 19, 2016 at 10:32 PM, Frank Wang <frankw@mit.edu> wrote:
On 12 January 2017 at 09:08, Frank Wang <frankw@mit.edu> wrote:
It looks like, if an app-level exception is in progress, w_result may be None. Try checking for that before setting the flags. Also: did you mean to alter w_1's rbflags? Seems strange that you'd alter w_1 but not w_2. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement.
Hi William, Good catch! I was just trying to get something to work, but it seems like you found another bug. I am now checking if it is None: # union flags in w_1 and w_2 and propagate to result if isinstance(w_result, W_Root) and w_result is not None: rb_results = {} if isinstance(w_1, W_Root) and w_1 is not None: w_1_rbflags = w_1.get_rbflags() rb_results.update(w_1_rbflags) if isinstance(w_2, W_Root) and w_2 is not None: w_2_rbflags = w_2.get_rbflags() rb_results.update(w_2_rbflags) w_result.set_rbflags(rb_results) I now get this error: [translation:ERROR] UnionError: Offending annotations: SomeInstance(can_be_None=False, classdef=pypy.interpreter.baseobjspace.W_Root) SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) Occurred processing the following simple_call: <MethodDesc 'set_rbflags' of <ClassDef 'pypy.interpreter.baseobjspace.W_Root'> bound to <ClassDef 'pypy.interpreter.baseobjspace.W_Root'> {}> returning v831 = simple_call(v830, rb_results_0) In <FunctionGraph of (pypy.interpreter.pyopcode:40)opcode_impl_for_div__star_2 at 0x1c190b10>: Happened at file /home/ubuntu/pypy2-v5.3.1-src/pypy/interpreter/pyopcode.py line 55 ==> w_result.set_rbflags(rb_results) Known variable annotations: v830 = SomePBC(can_be_None=False, descriptions={...1...}, knowntype=method, subset_of=None) rb_results_0 = SomeOrderedDict(dictdef=<{SomeImpossibleValue(): SomeImpossibleValue()}>) Processing block: block@203 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> in (pypy.interpreter.pyopcode:40)opcode_impl_for_div__star_2 containing the following operations: v830 = getattr(w_result_0, ('set_rbflags')) v831 = simple_call(v830, rb_results_0) --end-- I assume this happens because w_result can be a "subset" of None. What is the proper way to check if it's None? Frank On Wed, Jan 11, 2017 at 3:00 PM, William ML Leslie < william.leslie.ttg@gmail.com> wrote:
Hi, On 12 January 2017 at 03:35, Frank Wang <frankw@mit.edu> wrote:
Good catch! I was just trying to get something to work, but it seems like you found another bug. I am now checking if it is None:
No, I think these latest changes don't do anything. The error message says: you are storing in the rb_flags attribute something that is sometimes a dict, and sometimes an instance of W_Root. Check if there is somewhere else a subclass of W_Root with its own rb_flags attribute which would be set to a W_Root. Then you'd get a crash using that subclass, also untranslated, simply because 'rb_flags' is supposed to be two different things. To be on the safe side, you should change the name of the attribute in W_Root from rb_flags to something much more obscure like "_frank_rb_flags". A bientôt, Armin.
Re-hi, On 13 January 2017 at 12:34, Armin Rigo <armin.rigo@gmail.com> wrote:
says: you are storing in the rb_flags attribute something that is sometimes a dict, and sometimes an instance of W_Root.
Another potential reason would be that there is one buggy call somewhere that calls .set_rbflags(x) where x is an instance of W_Root instead of a dictionary. A bientôt, Armin.
Hi Armin, Thanks for the help! It seems like the problem is that somewhere in my code, I called .set_rbflags(x), and it's not checked that x is a dict. Frank On Fri, Jan 13, 2017 at 6:36 AM, Armin Rigo <armin.rigo@gmail.com> wrote:
participants (4)
-
Armin Rigo -
Carl Friedrich Bolz -
Frank Wang -
William ML Leslie