On Mon, 20 Feb 2012 23:23:13 +1000 Nick Coghlan wrote:

Does anyone object to me naming myself PEP czar for PEP 3144?

“Tsar is a title used to designate certain European Slavic monarchs or supreme rulers.” Is this our official word?

There's also basic-but-usable module documentation available (http://code.google.com/p/ipaddr-py/wiki/Using3144).

Mmmh, some comments: - a network can be "in" another network? Sounds strange. Compare with sets, which can be ordered, but not contained one within another. The idea of an address or network being "in" an interface sounds even stranger. - iterhosts()? Why not simply hosts()? - “A TypeError exception is raised if you try to compare objects of different versions or different types.”: I hope equality still works? Regards Antoine.

On Mon, Feb 20, 2012 at 14:23, Nick Coghlan wrote:

I don't personally think the module API needs the provisional disclaimer as the core functionality has been tested for years in ipaddr and the API changes in ipaddress are just cosmetic ones either for PEP 8 conformance, or to make the API map more cleanly to the underlying networking concepts. However, I'd be willing to include that proviso if anyone else has lingering concerns.

Should it be net.ipaddress instead of just ipaddress? Somewhat nested is better than fully flat. Cheers, Dirkjan

...

Does anyone object to me naming myself PEP czar for PEP 3144?

“Tsar is a title used to designate certain European Slavic monarchs or supreme rulers.”

Is this our official word?

"supreme ruler" sounds good to me. I could go for "inquisitor" instead of "czar" as well... Regards, Martin

On 20/02/2012 16:28, Senthil Kumaran wrote:

On Tue, Feb 21, 2012 at 12:07 AM, wrote:

...

"supreme ruler" sounds good to me. I could go for "inquisitor" instead of "czar" as well...

But that would be bad for developers from Spain as nobody would expect a spanish inquisition.

:-)

How about Big Brother then? As anyone worked in room 101? -- Cheers. Mark Lawrence.

On Mon, Feb 20, 2012 at 16:27, Antoine Pitrou wrote:

...

Should it be net.ipaddress instead of just ipaddress?

Somewhat nested is better than fully flat.

IMHO, nesting without a good, consistent, systematic categorization leads to very unpleasant results (e.g. "from urllib.request import urlopen").

Historically, our stdlib has been flat and I think it should stay so, short of redoing the whole hierarchy.

(note this has nothing to do with the possible implementation of modules as packages, such as unittest or importlib)

I thought Python 3 already came with a net package, but apparently that plan has long been discarded. So I retract my suggestion. Cheers, Dirkjan

On 2/20/2012 8:23 AM, Nick Coghlan wrote:

Does anyone object to me naming myself PEP czar for PEP 3144?

I think it great that you volunteer to be the PEP czar and hope Guido appoints you -- especially after your response to Antoine. Since this is a Python 3 module, let us start off with a modern Python 3 interface. That includes returning iterators instead of lists unless there is a really good reason. I can see how an outside developer could have difficulty getting integrated into our collective PEP process ;-). -- Terry Jan Reedy

Nick Coghlan wrote:

On Mon, Feb 20, 2012 at 11:55 PM, Antoine Pitrou wrote:

...

On Mon, 20 Feb 2012 23:23:13 +1000 Nick Coghlan wrote:

...

Does anyone object to me naming myself PEP czar for PEP 3144? “Tsar is a title used to designate certain European Slavic monarchs or supreme rulers.”

Is this our official word?

PEP czar/tsar and BDFOP (Benevolent Dictator for One PEP) are the two names I've seen for the role. I don't have a strong preference either way (just a mild preference for 'czar').

Also, "Czar" is commonly used in US politics as an informal term for the top official responsible for an area. "Drug Czar" is only the most familiar: http://en.wikipedia.org/wiki/List_of_U.S._executive_branch_%27czars%27 -- Steven

On Tue, Feb 21, 2012 at 7:09 AM, Guido van Rossum wrote:

Approved. Nick is PEP czar for PEP 3144. Thanks Nick!

In that case the addition of the "ipaddress" module is approved for 3.3, with a provisional caveat on the API details. I'm doing it that way because I think those remaining details can be better flushed out by the integration process (in particular, populating full module API reference documentation) than they could by another round of updates on the PEP and the ipaddr 3144 branch. At the very least: - the IP Interface API needs to move to a point where it more clearly *is* an IP Address and *has* an associated IP Network (rather than being the other way around) - IP Network needs to behave more like an ordered set of sequential IP Addresses (without sometimes behaving like an Address in its own right) - iterable APIs should consistently produce iterators (leaving users free to wrap list() around the calls if they want the concrete realisation) Initial maintainers will be me (for the semantically cleaner incarnation of the module API) and Peter (for the IPv4 and IPv6 correctness heavy lifting and ensuring any API updates only change the spelling of particular operations, such as adding a ".network." to some current operations on Interface objects, rather than reducing overall module functionality). This approach means we will still gain the key benefits of using the PyPI-tested ipaddr as a base (i.e. correct IP address parsing and generation, full coverage of the same set of supported operations) while exposing a simpler semantic model for new users that first encounter these concepts through the standard library module documentation: - IP Address as the core abstraction - IP Network as a container for IP Addresses - IP Interface as an IP Address with an associated IP Network Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 2/20/2012 11:52 PM, Guido van Rossum wrote:

On Mon, Feb 20, 2012 at 4:53 PM, Stephen J. Turnbull wrote:

...

Steven D'Aprano writes:

...

Also, "Czar" is commonly used in US politics as an informal term for the top official responsible for an area.

I think here the most important connotation is that in US parlance a "czar" does not report to a committee, and with the exception of a case where Sybil is appointed czar, cannot bikeshed. Decisions get made (what a concept!)

I'm curious how old that usage is. I first encountered it around '88 when I interned for a summer at DEC SRC (long since subsumed into HP Labs); the person in charge of deciding a particular aspect of their software or organization was called a czar, e.g. the documentation czar.

In US politics, the first I remember was the Drug Czar about that time. It really came into currently during Clinton's admin. -- Terry Jan Reedy

Just checking in: On Mon, Feb 20, 2012 at 5:48 PM, Nick Coghlan wrote:

At the very least: - the IP Interface API needs to move to a point where it more clearly *is* an IP Address and *has* an associated IP Network (rather than being the other way around)

This is done [1]. There's cleanup that needs to happen here, but the interface classes are now subclasses of the respective address classes. Now I need to apply some consistency and then move on to the remaining issues points:

- IP Network needs to behave more like an ordered set of sequential IP Addresses (without sometimes behaving like an Address in its own right) - iterable APIs should consistently produce iterators (leaving users free to wrap list() around the calls if they want the concrete realisation)

Cheers, peter [1] http://code.google.com/p/ipaddress-py/source/detail?r=10dd6a68139fb991162198... (the date is munged b/c I rebased to my original commit before submitting). -- Peter Moody      Google    1.650.253.7306 Security Engineer  pgp:0xC3410038

On Wed, Feb 29, 2012 at 9:13 PM, Peter Moody wrote:

Just checking in:

On Mon, Feb 20, 2012 at 5:48 PM, Nick Coghlan wrote:

...

At the very least: - the IP Interface API needs to move to a point where it more clearly *is* an IP Address and *has* an associated IP Network (rather than being the other way around)

This is done [1]. There's cleanup that needs to happen here, but the interface classes are now subclasses of the respective address classes.

Now I need to apply some consistency and then move on to the remaining issues points:

...

- IP Network needs to behave more like an ordered set of sequential IP Addresses (without sometimes behaving like an Address in its own right)

This is done [2]. Consistent iterable apis and polish left to do. Cheers, peter [2] http://code.google.com/p/ipaddress-py/source/detail?r=578ef1777018211f536cac...

...

- iterable APIs should consistently produce iterators (leaving users free to wrap list() around the calls if they want the concrete realisation)

Cheers, peter

[1] http://code.google.com/p/ipaddress-py/source/detail?r=10dd6a68139fb991162198... (the date is munged b/c I rebased to my original commit before submitting).

-- Peter Moody      Google    1.650.253.7306 Security Engineer  pgp:0xC3410038

-- Peter Moody      Google    1.650.253.7306 Security Engineer  pgp:0xC3410038

On Mon, Mar 19, 2012 at 12:44 PM, Peter Moody wrote:

On Mon, Mar 12, 2012 at 9:15 AM, Peter Moody wrote:

...

...

...

- iterable APIs should consistently produce iterators (leaving users free to wrap list() around the calls if they want the concrete realisation)

I might've missed earlier discussion somewhere, but can someone point me at an example of an iteratable api in ipaddr/ipaddress where an iterator isn't consistently produced?

There was at least one that I recall, now to find it again... And searching for "list" in the PEP 3144 branch source highlights subnet() vs iter_subnets() as the main problem child: https://code.google.com/p/ipaddr-py/source/browse/branches/3144/ipaddress.py... A single "subnets()" method that produced the iterator would seem to make more sense (with a "list()" call wrapped around it when the consumer really wants a concrete list). There are a few other cases that produce a list that are less clearcut. I *think* summarising the address range could be converted to an iterator, since the "networks" accumulation list doesn't get referenced by the summarising algorithm. Similarly, there doesn't appear to be a compelling reason for "address_exclude" to produce a concrete list (I also noticed a couple of "assert True == False" statements in that method for "this should never happen" code branches. An explicit "raise AssertionError" is a better way to handle such cases, so the code remains present even under -O and -OO) Collapsing the address list has to build the result list anyway to actually handle the deduplication part of its job, so returning a concrete list makes sense in that case. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote:

Nick Coghlan wrote:

...

Collapsing the address list has to build the result list anyway to actually handle the deduplication part of its job, so returning a concrete list makes sense in that case.

Having only one function return a list instead of an iterator seems questionable.

Depending on the code it could either keep track of what it has returned so far in a set and avoid duplication that way; or, just return an `iter(listobject)` instead of `listobject`.

I know I'm lacking context, but is the list ever expected to be huge? If not, what's wrong with always returning a list? -- --Guido van Rossum (python.org/~guido)

On Mon, Mar 19, 2012 at 1:13 PM, Ethan Furman wrote:

Guido van Rossum wrote:

...

On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote:

...

Nick Coghlan wrote:

...

Collapsing the address list has to build the result list anyway to actually handle the deduplication part of its job, so returning a concrete list makes sense in that case.

Having only one function return a list instead of an iterator seems questionable.

Depending on the code it could either keep track of what it has returned so far in a set and avoid duplication that way; or, just return an `iter(listobject)` instead of `listobject`.

I know I'm lacking context, but is the list ever expected to be huge? If not, what's wrong with always returning a list?

Nothing wrong in and of itself.  It just seems to me that if we have several functions that deal with ip addresses/networks/etc, and all but one return iterators, that one is going to be a pain... 'Which one returns a list again? Oh yeah, that one.'

It depends on whether they really are easy to confuse. If they are, indeed that feels like poor API design. But sometimes the only time two things seem confusingly similar is when you have not actually tried to use them. A naming convention often helps too.

Granted it's mostly a stylistic preference for consistency.

And remember that consistency is good in moderation, but if it becomes a goal in itself you may have a problem. -- --Guido van Rossum (python.org/~guido)

On Mon, Mar 19, 2012 at 12:55 PM, Guido van Rossum wrote:

On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote:

...

Nick Coghlan wrote:

...

Collapsing the address list has to build the result list anyway to actually handle the deduplication part of its job, so returning a concrete list makes sense in that case.

Having only one function return a list instead of an iterator seems questionable.

Depending on the code it could either keep track of what it has returned so far in a set and avoid duplication that way; or, just return an `iter(listobject)` instead of `listobject`.

I know I'm lacking context, but is the list ever expected to be huge? If not, what's wrong with always returning a list?

It's possible to return massive lists, (eg, returning the 4+ billion /128 subnets in /96 or something even larger, but I don't think that's very common). I've generally tried to avoid confusion by having 'iter' in the iterating methods, but if more of the methods return iterators, maybe I need to rethink that?

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/pmoody%40google.com

-- Peter Moody      Google    1.650.253.7306 Security Engineer  pgp:0xC3410038

On 3/19/2012 6:04 PM, Guido van Rossum wrote:

On Mon, Mar 19, 2012 at 2:58 PM, Peter Moody wrote:

...

On Mon, Mar 19, 2012 at 12:55 PM, Guido van Rossum wrote:

...

On Mon, Mar 19, 2012 at 12:37 PM, Ethan Furman wrote:

...

Nick Coghlan wrote:

...

Collapsing the address list has to build the result list anyway to actually handle the deduplication part of its job, so returning a concrete list makes sense in that case.

Having only one function return a list instead of an iterator seems questionable.

Depending on the code it could either keep track of what it has returned so far in a set and avoid duplication that way; or, just return an `iter(listobject)` instead of `listobject`.

I know I'm lacking context, but is the list ever expected to be huge? If not, what's wrong with always returning a list?

It's possible to return massive lists, (eg, returning the 4+ billion /128 subnets in /96 or something even larger, but I don't think that's very common). I've generally tried to avoid confusion by having 'iter' in the iterating methods, but if more of the methods return iterators, maybe I need to rethink that?

I personally like having 'iter' in the name (e.g. iterkeys() -- note that we dropped this in Py3k because it's no longer an iterator, it's a dict view now. But I don't want to promote that style for ipaddr.py.

I am not sure which way you are pointing, but the general default in 3.x is to return iterators: range, zip, enumerate, map, filter, reversed, open (file objects), as well at the dict methods. I am quite happy to be rid of the 'iter' prefix on the latter. This is aside from itertools. The main exceptions I can think of are str.split and sorted. For sorted, a list *must* be constructed anyway, so might as well return it. This apparently matches the case under consideration. If name differentiation is wanted, call it xxxlist. -- Terry Jan Reedy

On Tue, Mar 20, 2012 at 8:34 AM, Guido van Rossum wrote:

Anyway, I also tried to imply that it matters if the number of list items would ever be huge. It seems that is indeed possible (even if not likely) so I think iterators are useful.

But according to Nick's post, there's some sort of uniquification that is done, and the algorithm currently used computes the whole list anyway. I suppose that one could do the uniquification lazily, or find some other way to avoid that computation. Is it worth it to optimize an unlikely case?

On Mon, Mar 19, 2012 at 02:50:22PM -0700, Ethan Furman wrote:

Guido van Rossum wrote: [...]

...

And remember that consistency is good in moderation, but if it becomes a goal in itself you may have a problem.

While I agree that consistency as a goal in and of itself is not good, I consider it more important than 'moderation' implies; in my own code I try to only be inconsistent when there is a good reason to be.

I think we're probably in violent agreement, but I would put it this way: Consistency for its own sake *is* good, since consistency makes it easier for people to reason about the behaviour of functions on the basis that they are similar to other functions. But it is not the *only* good, and it is legitimate to trade-off one good for another good as needed.

To me, "it's already a list" isn't a good reason -- yes, that's easier for the library author, but is it easier for the library user? What is the library user gaining by having a list returned instead of an iterator?

I guess this discussion really hinges on which of these two positions you take: 1. The function naturally returns a list, should we compromise that simplicity by returning an iterator to be consistent with the other related/similar functions in the library? 2. These related/similar functions naturally return iterators, should we compromise that consistency by allowing one of them to return a list as it simplifies the implementation?

Of course, the flip-side also holds: what is the library user losing by getting an iterator when a list was available?

When we way the pros and cons, and it comes down to a smidgeon of performance in trade for consistency [1], I would vote for consistency.

I lean that way as well.

~Ethan~

[1] I'm assuming that 'iter(some_list)' is a quick operation.

For very small lists, it's about half as expensive as creating the list in the first place: steve@runes:~$ python3.2 -m timeit -s "x = (1,2,3)" "list(x)" 1000000 loops, best of 3: 0.396 usec per loop steve@runes:~$ python3.2 -m timeit -s "x = (1,2,3)" "iter(list(x))" 1000000 loops, best of 3: 0.614 usec per loop For large lists, it's approximately free: steve@runes:~$ python3.2 -m timeit -s "x = (1,2,3)*10000" "list(x)" 10000 loops, best of 3: 111 usec per loop steve@runes:~$ python3.2 -m timeit -s "x = (1,2,3)*10000" "iter(list(x))" 10000 loops, best of 3: 111 usec per loop On the other hand, turning the list iterator into a list again is probably not quite so cheap. -- Steven

Greg Ewing wrote:

Guido van Rossum wrote:

...

I personally like having 'iter' in the name (e.g. iterkeys() -- note that we dropped this in Py3k because it's no longer an iterator, it's a dict view now. But I don't want to promote that style for ipaddr.py.

+1 from me too on having all methods that return iterators clearly indicating so. It's an important distinction, and it can be very confusing if some methods of an API return iterators and others don't with no easy way of remembering which is which.

With the prevalence of iterators in Python 3 [1], the easy way is to have the API default to iterators, drop 'iter' from the names, and use 'list' in the names to signal the oddball cases where a list is returned instead. ~Ethan~ [1] http://mail.python.org/pipermail/python-dev/2012-March/117815.html