--- Исходное сообщение ---

От кого: "Alan G Isaac" Дата: 15 марта 2013, 20:38:38

On 3/15/2013 9:21 AM, Dmitrey wrote:

Temporary walkaround for a serious bug in FuncDesigner automatic differentiation kernel due to a bug in some versions of Python or NumPy,

Are the suspected bugs documented somewhere? the suspected bugs are not documented yet, I guess it will be fixed in future versions of Python or numpy the bug is hard to locate and isolate, it looks like this: derivative_items = list(pointDerivative.items()) # temporary walkaround for a bug in Python or numpy derivative_items.sort(key=lambda elem: elem[0]) ###################################### for key, val in derivative_items: indexes = oovarsIndDict[key] # this line is not reached in the involved buggy case if not involveSparse and isspmatrix(val): val = val.A if r.ndim == 1: r[indexes[0]:indexes[1]] = val.flatten() if type(val) == ndarray else val else: # this line is not reached in the involved buggy case r[:, indexes[0]:indexes[1]] = val if val.shape == r.shape else val.reshape((funcLen, prod(val.shape)/funcLen)) so, pointDerivative is Python dict of pairs (F_i, N_i), where F_i are hashable objects, and even for the case when N_i are ordinary scalars (they can be numpy arrays or scipy sparse matrices) results of this code are different wrt was or was not derivative_items.sort() performed; total number of nonzero elements is same for both cases. oovarsIndDict is dict of pairs (F_i, (n_start_i, n_end_i)), and for the case N_i are all scalars for all i n_end_i = n_start_i - 1.  Alan PS The word 'banausic' is very rare in English. Perhaps you meant 'unsophisticated'? google translate tells me "banausic" is more appropriate translation than "unsophisticated" for the sense I meant (those frameworks are aimed on modelling only numerical optimization problems, while FuncDesigner is suitable for modelling of systems of linear, nonlinear, ordinary differential equations, eigenvalue problems, interval analysis and much more). D.

On 3/15/2013 3:34 PM, Dmitrey wrote:

the suspected bugs are not documented yet

I'm going to guess that the state of the F_i changes when you use them as keys (i.e., when you call __le__. It is very hard to imagine that this is a Python or NumPy bug. Cheers, Alan

Hi, Different objects can have the same hash, so it compares to find the actual correct object. Usually when you store something in a dict and later you can't find it anymore, it is that the internal state changed and that the hash is not the same anymore. Matthieu 2013/3/16 Dmitrey

--- Исходное сообщение --- От кого: "Alan G Isaac" Дата: 15 марта 2013, 22:54:21

On 3/15/2013 3:34 PM, Dmitrey wrote:

...

the suspected bugs are not documented yet

I'm going to guess that the state of the F_i changes when you use them as keys (i.e., when you call __le__.

no, their state doesn't change for operations like __le__ . AFAIK searching Python dict doesn't calls __le__ on the object keys at all, it operates with method .__hash__(), and latter returns fixed integer numbers assigned to the objects earlier (at least in my case).

It is very hard to imagine that this is a Python or NumPy bug.

Cheers, Alan

_______________________________________________ NumPy-Discussion mailing listNumPy-Discussion@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/

Even if they have different hashes, they can be stored in the same underlying list before they are retrieved. Then, an actual comparison is done to check if the given key (i.e. object instance, not hash) is the same as one of the stored keys. 2013/3/16 Dmitrey

--- Исходное сообщение --- От кого: "Matthieu Brucher" Дата: 16 марта 2013, 11:33:39

Hi,

Different objects can have the same hash, so it compares to find the actual correct object. Usually when you store something in a dict and later you can't find it anymore, it is that the internal state changed and that the hash is not the same anymore.

my objects (oofuns) definitely have different __hash__() results - it's just integers 1,2,3 etc assigned to the oofuns (stored in oofun._id field) when they are created.

D.

Matthieu

2013/3/16 Dmitrey

--- Исходное сообщение --- От кого: "Alan G Isaac" Дата: 15 марта 2013, 22:54:21

On 3/15/2013 3:34 PM, Dmitrey wrote:

...

the suspected bugs are not documented yet

I'm going to guess that the state of the F_i changes when you use them as keys (i.e., when you call __le__.

no, their state doesn't change for operations like __le__ . AFAIK searching Python dict doesn't calls __le__ on the object keys at all, it operates with method .__hash__(), and latter returns fixed integer numbers assigned to the objects earlier (at least in my case).

It is very hard to imagine that this is a Python or NumPy bug.

Cheers, Alan

_______________________________________________ NumPy-Discussion mailing listNumPy-Discussion@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/

_______________________________________________ NumPy-Discussion mailing listNumPy-Discussion@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/

On 16 Mar 2013 11:49, "Dmitrey" wrote:

--- Исходное сообщение --- От кого: "Matthieu Brucher" Дата: 16 марта 2013, 12:39:07

...

Even if they have different hashes, they can be stored in the same

underlying list before they are retrieved. Then, an actual comparison is done to check if the given key (i.e. object instance, not hash) is the same as one of the stored keys.

...

but, as I have already mentioned, comparison of oofun(s) via __le__,

__eq__ etc doesn't change their inner state (but the methods can create additional oofun(s), although).

I have checked via debugger - my methods __le__, __eq__, __lt__, __gt__, __ge__ are not called from the buggy place of code, only __hash__ is called from there. Python could check key objects equivalence via id(), although, but I don't see any possible bug source from using id().

Dict lookup always calls both __hash__ and __eq__. I guess it might use id() to shortcut the __eq__ call in some cases - there are some places in python that do. Anyway there's no point trying to debug this code by ESP... It's not even clear from what's been said whether dict lookups have anything to do with the problem. -n

--- Исходное сообщение --- От кого: "Robert Kern" Дата: 16 марта 2013, 19:54:51 On Sat, Mar 16, 2013 at 10:39 AM, Matthieu Brucher < matthieu.brucher@gmail.com > wrote:

Even if they have different hashes, they can be stored in the same underlying list before they are retrieved. Then, an actual comparison is done to check if the given key (i.e. object instance, not hash) is the same as one of the stored keys.

Right. And the rule is that if two objects compare equal, then they must also hash equal. Unfortunately, it looks like `oofun` objects do not obey this property. oofun.__eq__() seems to return a Constraint rather than a bool, so oofun objects should simply not be used as dictionary keys. It is one of several base features FuncDesigner is build on and is used extremely often and wide; then whole FuncDesigner would work incorrectly while it is used intensively and solves many problems better than its competitors. That's quite possibly the source of the bug. Or at least, that's a bug that needs to get fixed first before attempting to debug anything else or attribute bugs to Python or numpy. Also, the lack of a bool-returning __eq__() will prevent proper sorting, which also seems to be used in the code snippet that Dmitrey showed. as I have already mentioned, I ensured via debugger that my __eq__, __le__ etc are not involved from the buggy place of the code, only __hash__ is involved from there.  -- Robert Kern

--- Исходное сообщение --- От кого: "Robert Kern" Дата: 16 марта 2013, 22:15:07 On Sat, Mar 16, 2013 at 6:19 PM, Dmitrey < tmp50@ukr.net > wrote:

--- Исходное сообщение --- От кого: "Robert Kern" < robert.kern@gmail.com > Дата: 16 марта 2013, 19:54:51

On Sat, Mar 16, 2013 at 10:39 AM, Matthieu Brucher < matthieu.brucher@gmail.com > wrote:

...

Even if they have different hashes, they can be stored in the same underlying list before they are retrieved. Then, an actual comparison is done to check if the given key (i.e. object instance, not hash) is the same as one of the stored keys.

Right. And the rule is that if two objects compare equal, then they must also hash equal. Unfortunately, it looks like `oofun` objects do not obey this property. oofun.__eq__() seems to return a Constraint rather than a bool, so oofun objects should simply not be used as dictionary keys.

It is one of several base features FuncDesigner is build on and is used extremely often and wide; then whole FuncDesigner would work incorrectly while it is used intensively and solves many problems better than its competitors.

I understand. It just means that you can't oofun objects as dictionary keys. Adding a __hash__() method is not enough to make that work. No, it just means I had mapped, have mapped, map and will map oofun objects as Python dict keys. As for the bug, I have found and fixed its source (I used some info from sorted list of free variables and somew other info from a non-sorted dict of oofun sizes). D.

--- Исходное сообщение --- От кого: "Robert Kern" Дата: 9 апреля 2013, 14:29:43

On Tue, Apr 9, 2013 at 4:15 PM, Dmitrey < tmp50@ukr.net > wrote:

--- Исходное сообщение --- От кого: "Robert Kern" < robert.kern@gmail.com > Дата: 16 марта 2013, 22:15:07

On Sat, Mar 16, 2013 at 6:19 PM, Dmitrey < tmp50@ukr.net > wrote:

...

--- Исходное сообщение --- От кого: "Robert Kern" < robert.kern@gmail.com > Дата: 16 марта 2013, 19:54:51

On Sat, Mar 16, 2013 at 10:39 AM, Matthieu Brucher < matthieu.brucher@gmail.com > wrote:

...

Even if they have different hashes, they can be stored in the same underlying list before they are retrieved. Then, an actual comparison is done to check if the given key (i.e. object instance, not hash) is the same as one of the stored keys.

Right. And the rule is that if two objects compare equal, then they must also hash equal. Unfortunately, it looks like `oofun` objects do not obey this property. oofun.__eq__() seems to return a Constraint rather than a bool, so oofun objects should simply not be used as dictionary keys.

It is one of several base features FuncDesigner is build on and is used extremely often and wide; then whole FuncDesigner would work incorrectly while it is used intensively and solves many problems better than its competitors.

I understand. It just means that you can't oofun objects as dictionary keys. Adding a __hash__() method is not enough to make that work.

No, it just means I had mapped, have mapped, map and will map oofun objects as Python dict keys.

Well, it's your software. You are free to make it as buggy as you wish, I guess. Yes, and that's why each time I get a bugreport I immediately start working on it, so usually I have zero opened bugs, as now . It somewhat differs from your bugtracker , that has tens of opened bugs, and ~ half of them are hanging for years (also, half of them are mentioned as high and highest priority) . But it's definitely your right to keep it as buggy as you wish, as well! D.  -- Robert Kern

On 04/10/2013 10:31 AM, Robert Kern wrote: You think comparing tracked bug counts across different projects means anything? That's adorable. I admire your diligence at addressing the bugs that you do acknowledge. That was never in question. But refusing to acknowledge a bug is not the same thing as fixing a bug. You cannot use objects that do not have a valid __eq__() (as in, returns boolean True if and only if they are to be considered equivalent for the purpose of dictionary lookup, otherwise returns False) as dictionary keys. Your oofun object still violates this principle. As dictionary keys, you want them to use their `id` attributes to distinguish them, but their __eq__() method still just returns another oofun with the default object.__nonzero__() implementation. This means that bool(some_oofun == other_oofun) is always True regardless of the `id` attributes. You have been unfortunate enough to not run into cases where this causes a problem yet, but the bug is still there, lurking, waiting for a chance hash collision to silently give you wrong results. That is the worst kind of bug. -- Robert Kern I had encountered the bugs with bool(some_oofun == other_oofun) when it was raised from other, than dict, cases, e.g. from "in list" (e.f. "if my_oofun in freeVarsList") etc, and had fixed them all. But that one doesn't occur from "in dict", I traced it with both debugger and putting print("in __eq__"),print("in __le__"), print("in __lt__"), print('in __gt__'), print('in __ge__') statements. As I had mentioned, removing mapping oofuns as dict keys is mere impossible - this is fundamental thing whole FuncDesigner is build on, as well as its user API. D.

On Wed, 2013-04-10 at 11:45 +0200, Sebastian Berg wrote:

On Wed, 2013-04-10 at 11:54 +0300, Dmitrey wrote:

...

On 04/10/2013 10:31 AM, Robert Kern wrote: <snip>

This is all good and nice, but Robert is still right. For dictionaries to work predictable you need to ensure two things.

First:

if object1 == object2: assert bool(hash(object1) == hash(object2))

and second, which is your case for the dictionary lookup to be predictable this must always work:

keys, values = zip(*dictionary.keys())

sorry dictionary.items() of course.

assert(keys.count(object2) == 1)

index = keys.index(object2) value = values[index]

And apparently this is not the case and it simply invites bugs which are impossible to track down because you will not see them in small tests. Instead you will have code that runs great for toy problems and can suddenly break down in impossible to understand ways when you have large problems.

So hopefully the list fixes you mention provide that the second code block will work as you would expect dictionary[object2] to work, but if this is not the case...

- Sebastian

...

As I had mentioned, removing mapping oofuns as dict keys is mere impossible - this is fundamental thing whole FuncDesigner is build on, as well as its user API. D.

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

An easy solution to all of this is to use a dict-like object that matches keys based on object identity while ignoring __hash__ and __eq__ entirely, e.g.: https://bitbucket.org/pypy/pypy/src/2f51f2142f7b/lib_pypy/identity_dict.py#c... -n On Wed, Apr 10, 2013 at 10:45 AM, Sebastian Berg wrote:

On Wed, 2013-04-10 at 11:54 +0300, Dmitrey wrote:

...

...

You think comparing tracked bug counts across different projects means anything? That's adorable. I admire your diligence at addressing the bugs that you do acknowledge. That was never in question. But refusing to acknowledge a bug is not the same thing as fixing a bug. You cannot use objects that do not have a valid __eq__() (as in, returns boolean True if and only if they are to be considered equivalent for the purpose of dictionary lookup, otherwise returns False) as dictionary keys. Your oofun object still violates this principle. As dictionary keys, you want them to use their `id` attributes to distinguish them, but their __eq__() method still just returns another oofun with the default object.__nonzero__() implementation. This means that bool(some_oofun == other_oofun) is always True regardless of the `id` attributes. You have been unfortunate enough to not run into cases where this causes a problem yet, but the bug is still there, lurking, waiting for a chance hash collision to silently give you wrong results. That is the worst kind of bug. -- Robert Kern I had encountered the bugs with bool(some_oofun == other_oofun) when it was raised from other, than dict, cases, e.g. from "in list" (e.f. "if my_oofun in freeVarsList") etc, and had fixed them all. But that one doesn't occur from "in dict", I traced it with both debugger and

On 04/10/2013 10:31 AM, Robert Kern wrote: putting print("in __eq__"),print("in __le__"), print("in __lt__"), print('in __gt__'), print('in __ge__') statements.

This is all good and nice, but Robert is still right. For dictionaries to work predictable you need to ensure two things.

First:

if object1 == object2: assert bool(hash(object1) == hash(object2))

and second, which is your case for the dictionary lookup to be predictable this must always work:

keys, values = zip(*dictionary.keys()) assert(keys.count(object2) == 1)

index = keys.index(object2) value = values[index]

And apparently this is not the case and it simply invites bugs which are impossible to track down because you will not see them in small tests. Instead you will have code that runs great for toy problems and can suddenly break down in impossible to understand ways when you have large problems.

So hopefully the list fixes you mention provide that the second code block will work as you would expect dictionary[object2] to work, but if this is not the case...

- Sebastian

...

As I had mentioned, removing mapping oofuns as dict keys is mere impossible - this is fundamental thing whole FuncDesigner is build on, as well as its user API. D.

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

--- Исходное сообщение --- От кого: "Alan G Isaac" Дата: 10 апреля 2013, 15:12:07 On 4/10/2013 3:31 AM, Robert Kern wrote:

You cannot use objects that do not have a valid __eq__() (as in, returns boolean True if and only if they are to be considered equivalent for the purpose of dictionary lookup, otherwise returns False) as dictionary keys. Your oofun object still violates this principle. As dictionary keys, you want them to use their `id` attributes to distinguish them, but their __eq__() method still just returns another oofun with the default object.__nonzero__() implementation. This means that bool(some_oofun == other_oofun) is always True regardless of the `id` attributes. You have been unfortunate enough to not run into cases where this causes a problem yet, but the bug is still there, lurking, waiting for a chance hash collision to silently give you wrong results. That is the worst kind of bug.

Hi Dmitrey, Robert and Sebastien have taken their time to carefully explain to your why your design is flawed. Your response has been only that you rely on this design flaw and it has not bitten you yet. It had bitten me some times till I understood the bugs source, but as I had mentioned I had fixed all those parts of code.  I trust you can see that this is truly not a response. The right response is to explore how you can refactor to eliminate this lurking bug, or to prove that it can *never* bite due to another design feature. You have done neither, and the second looks impossible. So you have work to do.  You say that you *must* use oofuns as dict keys. This is probably false, but you clearly want to retain this aspect of your design. But this choice has an implication for the design of oofuns, as carefully explained in this thread. So you will have to change the design, even though that may prove painful. Refactoring is mere impossible, user API and thouzands lines of whole FuncDesigner kernel heavily relies on the oofuns as dict keys. Also, I don't see any alternative that is as convenient and fast as the involved approach. As for new features, I just keep it in mind while implementing them, and now it's quite simple.  No smaller step is adequate to the quality of software you aspire to. One last thing. When someone like Robert or Sebastien take their time to explain a problem to you, the right response is "thank you", even if their news is unwelcome. Don't shoot the messenger. I understand your opinion, but I'm not a kind of person who thanks on responses like "Well, it's your software. You are free to make it as buggy as you wish" (Robert have apologised, although). Also, I haven't thanked Sebastien because I was AFK. Thanks for all who participated in the thread. D.

On 10 Apr 2013 08:01, "Dmitrey" wrote:

--- Исходное сообщение --- От кого: "Robert Kern" Дата: 9 апреля 2013, 14:29:43

...

On Tue, Apr 9, 2013 at 4:15 PM, Dmitrey wrote:

...

--- Исходное сообщение --- От кого: "Robert Kern" Дата: 16 марта 2013, 22:15:07

On Sat, Mar 16, 2013 at 6:19 PM, Dmitrey wrote:

...

--- Исходное сообщение --- От кого: "Robert Kern" Дата: 16 марта 2013, 19:54:51

On Sat, Mar 16, 2013 at 10:39 AM, Matthieu Brucher wrote:

...

Even if they have different hashes, they can be stored in the same underlying list before they are retrieved. Then, an actual

...

...

...

...

done to check if the given key (i.e. object instance, not hash) is

...

...

...

...

same as one of the stored keys.

Right. And the rule is that if two objects compare equal, then they must also hash equal. Unfortunately, it looks like `oofun` objects do not obey this property. oofun.__eq__() seems to return a Constraint rather than a bool, so oofun objects should simply not be used as dictionary keys.

It is one of several base features FuncDesigner is build on and is used extremely often and wide; then whole FuncDesigner would work incorrectly while it is used intensively and solves many problems better than its competitors.

I understand. It just means that you can't oofun objects as dictionary keys. Adding a __hash__() method is not enough to make that work.

No, it just means I had mapped, have mapped, map and will map oofun objects as Python dict keys.

Well, it's your software. You are free to make it as buggy as you wish, I guess.

Yes, and that's why each time I get a bugreport I immediately start working on it, so usually I have zero opened bugs, as now . It somewhat differs from your bugtracker , that has tens of opened bugs, and ~ half of

comparison is the them are hanging for years (also, half of them are mentioned as high and highest priority) . But it's definitely your right to keep it as buggy as you wish, as well! This kind of personal attack is never appropriate for this list. Please stop. -n