Phillip J. Eby wrote:

At 04:52 PM 6/21/2006 -0400, Jim Fulton wrote:

...

I was going to say, of course, and give you a link, but I'm surprised to discover there's nothing I can find. Which is weird because I *know* I wrote something.

I thought you did too.

However I suspect it was originally on the PythonEggs wiki page, and it didn't get transferred to the PkgResources or setuptools pages before being deleted.

Ah, yes, that's in fact the case. Here's the missing paragraph, grabbed from the wiki history:

"""Requirement strings basically consist of a distribution name, an optional list of "options" (more on this in a moment), and a comma-separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of > or >= and < or <= conditions, or exactly matches a == or != condition."""

OK, this is something. I'm sure what "ascending version order" is. In ascending version order, what is the ordering of: <1.2, >1.2, ==1.2, <=1.2, >=1.2, and !=1.2 ? -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Jun 21, 2006, at 6:21 PM, Phillip J. Eby wrote:

At 05:36 PM 6/21/2006 -0400, Jim Fulton wrote:

...

In ascending version order, what is the ordering of:

<1.2, >1.2, ==1.2, <=1.2, >=1.2, and !=1.2

?

It's actually unspecified, because it makes no sense to have more than one condition for the same version. What could the assortment of values you just listed possibly *mean*?

I don't know. If it's illegal, then say so. The software` certainly accepts it now. I think it would be helpful to call this out in your specification. I also think it might be good to expand the explanation a bit, perhaps with some examples. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

At 07:17 AM 6/22/2006 -0400, Jim Fulton wrote:

On Jun 21, 2006, at 5:11 PM, Phillip J. Eby wrote: ...

...

"""Requirement strings basically consist of a distribution name, an optional list of "options" (more on this in a moment), and a comma- separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of > or >= and < or <= conditions, or exactly matches a == or != condition."""

I suppose that given:

<1.2, >1.3

there is an implicit >-infinity and 1,3 and

Yes.

At 12:26 PM 6/22/2006 -0400, Jim Fulton wrote:

On Jun 21, 2006, at 5:11 PM, Phillip J. Eby wrote:

...

"""Requirement strings basically consist of a distribution name, an optional list of "options" (more on this in a moment), and a comma- separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of > or >= and < or <= conditions, or exactly matches a == or != condition."""

I don't think this is right. :) ... When scanning left to right, 1.9 matches !=1.1, so it should match and, since it is the highest version, it should be returned. Either your description of the algorithm is incorrect or I'm misunderstanding it.

You're missing the "exactly matches" part. The relevant context is: "Until the package's version ... exactly matches a == or != condition". Perhaps making that "a == or != condition's version" would have been clearer, as that's what I meant by that phrase. Anyway, "1.9" does not exactly match the "1.1" in "!=1.1".

At 01:05 PM 6/22/2006 -0400, Jim Fulton wrote:

On Jun 22, 2006, at 12:49 PM, Phillip J. Eby wrote:

...

At 12:26 PM 6/22/2006 -0400, Jim Fulton wrote:

...

On Jun 21, 2006, at 5:11 PM, Phillip J. Eby wrote:

...

"""Requirement strings basically consist of a distribution name, an optional list of "options" (more on this in a moment), and a comma- separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of > or >= and < or <= conditions, or exactly matches a == or != condition."""

I don't think this is right. :) ... When scanning left to right, 1.9 matches !=1.1, so it should match and, since it is the highest version, it should be returned. Either your description of the algorithm is incorrect or I'm misunderstanding it.

You're missing the "exactly matches" part. The relevant context is: "Until the package's version ... exactly matches a == or != condition". Perhaps making that "a == or != condition's version" would have been clearer, as that's what I meant by that phrase.

Anyway, "1.9" does not exactly match the "1.1" in "!=1.1".

Um, OK. So I guess the idea is that we scan these things trying to make a decision. The decision is either match or not match. Is that how I was supposed to read the above quote?

Um, no. :) The specification is: 1. "The version conditions you supply are sorted into ascending version order" 2. "and then scanned left to right" (from low to high version number) 3. "until the package's version" 3a. "falls between a pair of > or >= and < or <= conditions" 3b. "or exactly matches a == or != condition" In both #3a and #3b, we are saying that the package's version is compared to the condition's version. If the version is *between* the version of a > or >= condition on the left, and the version of a < or <= condition on the right, then it is accepted. If the version exactly matches the version of a == or != condition, then it is either accepted (==) or rejected (!=). If neither #3a nor #3b happens, then we continue "scanning left to right" (per #2). I suspect that you will next come back and ask about this:

1.0, !=1.2, <2.0

at which point I will remind you that #3a says nothing about the pairs being adjacent. :)

On Jun 22, 2006, at 1:56 PM, Jim Fulton wrote:

I assume you are alluding to what I assume is the case that that there is an implicit

...

-infinity and

No, that can't be right. If that were so, then ==1.2 would be meaningless So I have no idea what you were trying to say. The more I find out about the specification specification, the less I understand it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Jun 22, 2006, at 3:00 PM, Phillip J. Eby wrote:

At 02:20 PM 6/22/2006 -0400, Jim Fulton wrote:

...

On Jun 22, 2006, at 1:56 PM, Jim Fulton wrote:

...

I assume you are alluding to what I assume is the case that that there is an implicit

...

-infinity and

No, that can't be right. If that were so, then

==1.2 would be meaningless

So I have no idea what you were trying to say.

The infinities are implied only if they are needed to complete a pairing with an actual condition.

...

The more I find out about the specification specification, the less I understand it.

Exactly, so stop trying to understand it, and you'll see how obvious it is. :)

Ah, if only it was.

The implementation scans from left to right until it's *sure* that the version is either accepted or rejected. Each condition is a point, a bound, or a point and a bound. If the version matches the "point" part of condition, it's an exact match and you are *sure* of an accept, unless it's a "!=", in which case you're *sure* it's a reject.

If the version falls below an upper bound (< or <=), you are also *sure* that it's accepted. If it is above an upper bound, it is *tentatively* rejected.

So given: >1, <3, >5, <7 You are sure that 4 is accepted? Given: <2, <5 Is 3 accepted or rejected?

If the version falls below a lower bound (> or >=), you are *sure* that it's rejected. If it is above a lower bound, it is *tentatively* accepted.

So given: >1, <3, >5, <7 You are sure that 2 is rejected?

If you reach the end of the conditions without being "sure" of anything, then your most recent tentative acceptance or rejection is used.

A simple state machine is used to implement this:

state_machine = { # =>< '<' : '--T', '<=': 'T-T', '>' : 'F+F', '>=': 'T+F', '==': 'T..', '!=': 'F++', }

cmp() is used to determine whether the version is =, >, or < than the condition's version, and the appropriate row and column is pulled from the above table. "T" means "sure accept", "F" means "sure reject", "+" means "tentative accept", and "-" means "tentative reject". ("." means "don't care".) The state machine simply compares versions until its sure or there are no more to compare.

I don't understand the meaning of the values in the dictionary above. Do the character positions reflect states somehow? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Jun 22, 2006, at 3:44 PM, Phillip J. Eby wrote:

At 03:20 PM 6/22/2006 -0400, Jim Fulton wrote:

...

...

The implementation scans from left to right until it's *sure* that the version is either accepted or rejected. Each condition is a point, a bound, or a point and a bound. If the version matches the "point" part of condition, it's an exact match and you are *sure* of an accept, unless it's a "!=", in which case you're *sure* it's a reject.

If the version falls below an upper bound (< or <=), you are also *sure* that it's accepted. If it is above an upper bound, it is *tentatively* rejected.

So given: >1, <3, >5, <7 You are sure that 4 is accepted?

Scanning left to right, >1 is a lower bound and 4 is above it, so that's a tentative accept. Next we see <3, and it is an upper bound, and we fall below it, so it is a sure accept at that point, and we stop scanning.

Huh? 4 is below 3?

...

Given: <2, <5 Is 3 accepted or rejected?

<2, upper bound, 3 is above, tentative reject. <5, upper bound, 3 is below it, sure accept. (Intuitively -- at least for me -- <5 is paired with >-infinity here.)

OK. I think that many people would find this non-obvious.

...

...

If the version falls below a lower bound (> or >=), you are *sure* that it's rejected. If it is above a lower bound, it is *tentatively* accepted.

So given: >1, <3, >5, <7 You are sure that 2 is rejected?

...

1, lower bound, 2 is above, tentative accept. <3, upper bound, 2 is below, sure accept, scanning stops.

Ah, so in your explanation above, "f the version falls below a lower bound" only applies to a scanning position. OK, that clarifies this case.

...

...

A simple state machine is used to implement this:

state_machine = { # =>< '<' : '--T', '<=': 'T-T', '>' : 'F+F', '>=': 'T+F', '==': 'T..', '!=': 'F++', }

cmp() is used to determine whether the version is =, >, or < than the condition's version, and the appropriate row and column is pulled from the above table. "T" means "sure accept", "F" means "sure reject", "+" means "tentative accept", and "-" means "tentative reject". ("." means "don't care".) The state machine simply compares versions until its sure or there are no more to compare.

I don't understand the meaning of the values in the dictionary above. Do the character positions reflect states somehow?

It's a truth table: the rows are condition operators, and the columns are cmp() results. It is simply a transcription of the rules about points and bounds that I spelled out verbally, reduced to a table lookup on the condition and the comparison results.

OK, than makes sense. So, with the requirement: >1, <3, >5, <7 So let's see, with 4, we get +-F and we reject it. OK, that makes sense. The state machine helps a lot. My question is now answered. I think that the fact that you need to understand a non-trivial algorithm with a state machine to understand how non-trivial specifications are interpreted is a problem. Maybe it's enough to tell people "don't use complex specifications", but maybe it would be better to use a simpler system. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

At 04:12 PM 6/22/2006 -0400, Jim Fulton wrote:

I also think that the complete algorithm, including the state machine needs to be documented clearly. Your original paragraph really isn't adequate.

Only if you try to use it to interpret requirements that contain redundant information.

Maybe, you should document simple cases and refer to the full complex model for non-trivial cases.

If by non-trivial you mean cases that contain redundant information, sure. For example "<1,<2" is redundant, since "<1" implies "<2". Only the "<1" is needed, and my assumption was that a person will not write "<1, <2" unless there is also a ">" or ">=" somewhere between the "<1" and the "<2" (in ascending version order). Similarly, the only time multiple conditions for one version make sense is to have "<1,>1", but this is the same as "<>1" or "!=1". If you simply write the shortest possible requirement that expresses the version range(s) and exceptions that you do or don't want, you will find that the 1-paragraph explanation will provide you with an easy and unambiguous interpretation: scanning left to right, see if the actual version is in one of the allowed ranges, or matches a point condition. Translating this specification into an algorithm is tedious, but essentially a mechanical matter. All of your questions, on the other hand, have been assuming that there is a simple algorithm that translates into something not necessarily simple to explain. But it's the explanation that's simple, not the algorithm. Version specifications are just ranges and exceptions. Admittedly, it would have been better if Requirement had been designed to reject redundant input -- if only because it would've kept you from then asking what various redundant things *mean*. ;-) For the most part though, the implementation just ignores anything redundant.

I think you are making a mistake to assume that everyone has the same notion of obvious that you do. For example, someone could reasonably expect that ==1.0 matches 1.1.1. (I'm not advocating this interpretation, but simply pointing out that "obvious" is not universal.) In addition, people sometimes make typos. A system that figures out what they "really meant" rather than complain when they enter something that doesn't make sense isn't doing them any favors. In any case, I expect that having people build tools on top of setuptools is a use case you anticipated. For people to do that, they sometimes need precise specifications of behavior. Jim On Jun 22, 2006, at 4:36 PM, Phillip J. Eby wrote:

At 04:08 PM 6/22/2006 -0400, Jim Fulton wrote:

...

...

...

So given: >1, <3, >5, <7 You are sure that 4 is accepted?

Scanning left to right, >1 is a lower bound and 4 is above it, so that's a tentative accept. Next we see <3, and it is an upper bound, and we fall below it, so it is a sure accept at that point, and we stop scanning.

Huh? 4 is below 3?

Hm, somehow I got confused with 2. It should say, <3 is an upper bound, 4 is above it, tentative reject. >5 is a lower bound, 4 is below it, it's a sure reject. I think I copied 2 in from another example while editing.

...

...

...

Given: <2, <5 Is 3 accepted or rejected?

<2, upper bound, 3 is above, tentative reject. <5, upper bound, 3 is below it, sure accept. (Intuitively -- at least for me -- <5 is paired with >-infinity here.)

OK. I think that many people would find this non-obvious.

You keep missing the big picture. The algorithm is designed to interpret a *meaningful* list of versions *specified by a human*. Humans are usually not excessively redundant, and will thus tend to express a version requirement in the form of ranges with exceptions.

Humans tend to think that it is obvious that if they ask for a version <5, they do not need to also specify that the version be more than negative infinity, or even that it be greater than zero. It is, after all, a *version* number. :)

Similarly, if I ask for a version >10, I assume that you will understand I want it to be less than negative infinity. :)

If it weren't for the fact that '.' and '-' are often used in version numbers, I might have used range expressions using '-' or '...', but these are also harder to read for the most common case where you want '>=someversion'.

...

Ah, so in your explanation above, "f the version falls below a lower bound" only applies to a scanning position. OK, that clarifies this case.

If you look back at my previous descriptions, the concept of scanning is repeated many times, including the simple 1-paragraph description. It's all scanning, just like a human would do when reading a series of version requirements.

...

I think that the fact that you need to understand a non-trivial algorithm with a state machine to understand how non-trivial specifications are interpreted is a problem.

It's only a problem if you try to reduce it to an algorithm. If you simply try to *understand* a version requirement, it's really quite straightforward. Think of what the human who wrote the requirement is saying:

"Um, let's see, it should be less than version 3, because that version's got a new API, and it should be more than version 1.2, because there were some bugfixes, and oh yeah, don't use version 1.6 because it was just plain hosed. Oh, and 1.1.3 is also good."

The algorithm is designed to correctly interpret a person who is simply recording their wishes in this manner -- describing a series of acceptable or unacceptable ranges, and mentioning specific versions that are exceptions to the ranges.

...

Maybe it's enough to tell people "don't use complex specifications", but maybe it would be better to use a simpler system.

People don't normally express themselves redundantly -- especially programmers. The target users of specifications are these humans, not the programmers trying to interpret the specification for specifications. I did not anticipate that particular user and use case. ;)

-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Jun 22, 2006, at 6:51 PM, Phillip J. Eby wrote:

At 04:54 PM 6/22/2006 -0400, Jim Fulton wrote:

...

In any case, I expect that having people build tools on top of setuptools is a use case you anticipated. For people to do that, they sometimes need precise specifications of behavior.

Of course -- but I didn't anticipate that they would need anything more than the ability to test whether a version matches a requirement.

I think I've explained, and I thought you understood why that wasn't adequate for me. ...

Also, at some point, one does what one can with the available manpower. Bob Ippolito was the only person besides me who was actively involved in the design process at the time -- and he was kind of like, "whatever, man" on the subject. :) It would've been nice to have had your participation then. Now, we'll pretty much have to live with what we've got.

1. You should know that no system is perfect. You can't anticipate all requirements up front. Surely, you want people to use this system beyond those who invented it. It's also not practical for all of us to get involved in every design at it's inception. 2. I have tried not to ask you to implement my use cases. There are a number of reasons for this, including: - I don't know fully what my use cases are. After all, use cases change with experience. - I don't want to disrupt setuptools development. I don't think setuptools should try to meet everyone's needs. I do think that people should be able to meet their own needs on top of setuptools. I think setuptools has stood up very well in this regard. I really haven't asked for much. I believe the only changes you've made for me so far have been bug fixes. I don't intend to ask for much. The only non-bug-fix change I've asked for so far has been to declare public a pre-exiting API so that I can implement what I want myself. Of course, I've also asked that some semantics be documented. I needed this documentation for what I was doing. I also believe that documenting things well sheds important light on them. You learn things about decisions to make by explaining these decisions to other. I think setuptools is pretty cool and I have sung it's praises in many forums, admittedly, even before I know what I was talking about. ;) I don't regret that. I'm building a "buildout" system on top of setup tools that will help it meet use cases that it doesn't meet now. I appreciate your advice and support during this process process and look forward to more of the same in the future. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org