Raymond Hettinger <python@rcn.com> wrote:

They include [...] the str.partition function,

Where is the current version of this patch? After reharsing the archives, I have an additional suggestion which I didn't find already mentioned in the discussion. What about: str.partition() -> (left, right or None) str.rparition() -> (left or None, right) which means: "foo:bar".partition(":") -> ("foo", "bar") "foo:bar".rpartition(":") -> ("foo", "bar") "foo:".partition(":") -> ("foo", "") "foo:".rpartition(":") -> ("foo", "") ":foo".partition(":") -> ("", "foo") ":foo".rpartition(":") -> ("", "foo") "foo".partition(":") -> ("foo", None) "foo".rpartition(":") -> (None, "foo") Notice that None-checking can be done as a way to know if the separator was found. I mentally went through the diff here (http://mail.python.org/pipermail/python-dev/2005-August/055781.html) and found out that most (all?) the usages of '_' disappears with this semantic. -- Giovanni Bajo

[Raymond]

FYI, I have several non-major C components to go in but not in time for alpha 1. They include some minor fix-ups in the sets module, the str.partition function, add gc to itertools.tee, a couple of functions in binascii, add itertools.izip_longest, Crutcher's patch to make exec take dictionary arguments, move the peepholer to just before the assembler, and implement Alex's iterator copier for a number of iterables (xrange, repeat, count, reversed, list, tuple, deque, dict, and set). All of these have been previously discussed/approved and should go in to the second alpha.

The only one that is borderline is Crutcher's patch. I will try to free up some time to get that into alpha 1. It touches critical parts of the interpreted and needs to be reviewed, tested, timed, and exercised thoroughly.

[Neal]

All those seem reasonable. Do the functions you mention address Josiah's patch?

I believe so. They are binascii.b2long() and binascii.long2b(). And, no, the name of the latter wasn't taken from a song ;-) Raymond

On 3/18/06, Barry Warsaw <barry@python.org> wrote:

On Fri, 2006-03-17 at 23:48 -0800, Neal Norwitz wrote:

...

Just in case anybody here's been snoozing, 2.5 alpha 1 is coming up real quick, hopefully within a couple of weeks. If you have any *major* features (particularly implemented in C) that you want to see in 2.5, bring it up now. I want to strive for feature completeness by alpha 1. I know we will have some .py modules that won't make it into alpha 1, but they really should make it in by alpha 2 or be deferred to 2.6.

I'd like to get some feedback on my PEP 352 comments, and if there's general agreement on the hierarchy I proposed (so far so good :), then I'd like to take a crack at implementing them.

-1. See my response in the other thread. The focus on 'Error' is mistaken, and we have a large body of existing code that derives from Exception. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 3/19/06, Barry Warsaw <barry@python.org> wrote:

On Sun, 2006-03-19 at 19:45 -0800, Guido van Rossum wrote:

...

-1. See my response in the other thread. The focus on 'Error' is mistaken, and we have a large body of existing code that derives from Exception.

Just to be clear, are you saying -1 only for Python 2.5 or -1 also for Python 3.0? If the former, as I mentioned before, that would be fine with me. If the latter too, then I won't waste my time following up on the thread or writing a PEP.

I'm -0 for changing this in 3.0; a larger-scale reorganization could be undertaken but it's not a big priority. Before you spend more energy on this, I'd like to address the process for Python 3000, which is too chaotic right now. See my new thread titled "Python 3000 Process". -- --Guido van Rossum (home page: http://www.python.org/~guido/)

[Barry Warsaw]

...

Oh, also, we have a couple of additions to the PySet C API. I'll work on putting together an SF patch for them over the weekend.

What are you proposing to add to the PySet API? I designed an API that was both minimal and complete. The idea was to provide direct access to fined grained functions and access the rest through the existing abstract API for PyObject and PyNumber as detailed in the Set API docs. I tried out the API to translate a number of set algorithms and found that the API was easy-to-use and sufficient as-is. There may be room for variants of the type checking macros, but I would like the rest of the C API to remain as-is unless some compelling deficiency can be shown. It is easy to expand the API later but almost impossible to take anything back once in the field. IOW, if I have I still have a say in the matter, the patch will most likely not be accepted. Raymond

[Raymond]

...

What are you proposing to add to the PySet API?

[Barry]

PySet_Clear(), PySet_Next(), PySet_Update(), and PySet_AsList().

PySet_Clear() ------------- Use PyObject_CallMethod(s, "clear", NULL). Or if you need to save a millisecond on an O(n) operation, use PyNumber_InPlaceSubtract(s,s) as shown in the docs. If the name bugs you, it only takes a one-line macro to define a wrapper. The set API should not be cluttered with unnecessary and redundant functions. PySet_Next() ------------ This is also redundant. The preferred way to iterate over a set should be PyObject_GetIter(s). The iter api is generic and works for all containers. It ought to be the one-way-to-do-it. Further, it doesn't make sense to model this after the dictionary API where the next function is needed to avoid double lookups by returning pointers to both the key and value fields at the same time (allowing for modification of the value field). In contrast, for sets, there is no value field to look-up or mutate (the key should not be touched). So, we shouldn't be passing around pointers to the internal structure. I want to keep the internal structure of sets much more private than they were for dictionaries -- all access should be through the provided C API functions -- that keeps the implementation flexible and allows for future improvements without worrying that we've broken code for someone who has touched the internal structure directly. Also, the _Next() api is not as safe as the _GetIter api which checks for mutation during iteration. The safety should not be tossed aside without good reason. PySet_Update() --------------- Use PyObject_CallMethod(s, "update", "O", iterable). That is the preferred way to access all of the high volume methods. Only the fine grained methods (like contains, add, pop, or discard) have a need for a direct call. Adding unnecessary functions for the many-at-once methods gains you nothing -- perhaps saving a tiny O(1) look-up step in an O(n) operation. FWIW, the same reasoning also applies to why the list API defines PyList_Append() but not PyList_Extend(). PySet_AsList() --------------- There is already a function expressly for this purpose, PySequence_List(s). It is clear, readable, and is the one-way-to-do-it for turning arbitrary iterables into a list. FWIW, that function already has an optimization to pre-size the result list to the correct size, so it runs pretty fast (no over-allocate / resize dance). I had considered putting a further optimization inside PySequence_List to have a special case path for sets (like it does for tuples); however, it occurred to me that this can never be the time critical part of a program since the time to convert a set to a list is small compared to the time to construct the set in the first place (many times longer). IOW, further micro-optimization here is pointless. [Raymond]

...

IOW, if I have I still have a say in the matter, the patch will most likely not be accepted.

[Barry]

Can you explain why the additions above would not be obvious improvements?

Yes. A fatter api is not a better api. The PyObject_CallMethod() approach is highly readable and assures direct correspondence with a Python set method of the same name. Trying to save the method name lookup time on O(n) methods is a false optimization. The concrete API should avoid duplicating services provided by the abstract API such as PySequence_List(). Also, the set API should not model parts of the dict API that grant direct access to the internal structure or were needed because dictionaries had a value field. As it stands now, it is possible to use sets in C programs and access them in a way that has a direct correspondence to pure Python code -- using PyObject_CallMethod() for things we would usually access by name, using the PyNumber API for things we would access using operators, using other parts of the abstract API exactly as we would in Python (PyObject_Repr, PyObject_GetIter, PySequence_List, PyObject_Print, etc.), and using a handful of direct access functions for the fine grained methods like (add, pop, contains, etc.). IOW, I like the way the C code looks now and I like the minimal, yet complete API. Let's don't muck it up. FWIW, the C implementation in Py2.5 already provides nice speed-ups for many operations. Likewise, its memory requirements have been reduced by a third. Try to enjoy the improvements without gilding the lily. Cheers, Raymond

[Barry]

Is it your intent to push for more use of the abstract API instead of the concrete APIs for all of Python's C data structures? Current API aside, are you advocating this approach for all new built-in types? Would you argue that Python 3.0's C API be stripped of everything but the abstract API and the bare essentials of the concrete API?

It's not up to me. Perhaps someone else can chime-in about the philosophy of how the C API is supposed to balance abstract and concrete APIs. For concrete APIs, it does make sense to have methods for fine-grained access to containers (for the O(1) ops that tend to appear inside loops). I know that the more one uses the abstract API, the more likely the code is going to be able to accept duck typed inputs. Also, most things that have tp_slots have a corresponding abstract method instead of tons a concrete access points; hence, I would be supportive if you proposed a PyObject_Clear(o) function (for calling tp_clear slots when they exist and returning an error code when they don't). For setobject.c, if I still have a say in the matter, my strong preference is to keep the API minimal, expose fine-grained functions for efficiency, use PyNumber methods for direct access to operator style set operations, and use the abstract API for everything else. Though you have a different world view about fat vs thin APIs and on whether the C API should track the Python API, perhaps you can agree with my reasoning in this case. There is a semantic difference between code like s+=t and s.update(t). The former only works when t is a set and the latter works for any iterable. When the C code corresponds to the Python code, that knowledge is kept intact and there is no confusion between PyNumber_InPlaceAdd(s,t) vs PyObject_CallMethod(s, "update", "(O)", t). I did not want to fill-up the C API with methods corresponding to every named method in the Python API -- besides making the API unnecessarily fat, IMO, it introduces ambiguity about whether PySet_Update would be the set-only operation or the generator iteratable version. With respect to clear(), I do not not want to further enshrine that method. IMO, most uses of it are misguided and the code would be better-off allowing the existing set to be decreffed away and using PySet_New() instead. I realize that is a matter of taste, but it is a basic of API design that the module author provide clues about how the module is intended to be used. With respect to PySet_Next(), I do agree that it is both a bit faster and easier to use than the iterator API. That is more of a reflection of issues with the iterator API than it is an indication that PySet_Next() would be worthwhile. That being said, I stand by my rejection of that method. For dictionaries, the problems with the approach were some offset by the advantages of getting a simultaneous key/value lookup and by direct access to the memory where the value was stored. Sets, of course, do not have these offseting benefits. The problems with the approach however do apply to sets. I do not want to be passing around pointers to the private internal hash table -- programs have no business altering that memory directly. You have to be careful with even simple cases such as using PyString_AS_STRING which returns a pointer and then the underlying object can get decreffed away leaving a pointer to an invalid memory location (that happened to me recently but I caught it during code review). Also, there is the safety issue of having the table mutate during iteration. While your note downplayed the issue noting that the risks are fully disclosed, I speak from experience in saying that it is all too easy to accidentally allow the table to mutate during iteration -- I learned this the hard-way and had to undo a number of my uses of set_next() which had been used for speed and for being simpler than the iteator api. A single decref or call to a key's PyObject_Hash() is enough to trigger arbitrary Python code running during the middle of iteration, possibly resulting in a resize or value mutation. Take a look at the code for set_clear() to see the dance required to avoid this risk. IOW, the safety considerations alone preclude PySet_Next(). Instead, use PyObject_GetIter() and enjoy the benefits of safety, duck typing, and re-usable code. Because of the safety issues and passing internal pointers, I prefer that the _Next() api NOT get replicated throughout the language. It was needed for dicts, but shouldn't start sprouting up everywhere. All that being said, I understand that some of it comes down to taste and philosophical differences. As the module author, the existing API of course reflects my tastes and world-view. I have given you simple, one line alternatives to each proposal and listed the reasons for each choice, so it can't be argued that the API is somehow crippling your work. I'm sympathethic to your reluctance to use PyObject_CallMethod(). But, do understand that it is simply an aversion. It works just fine and makes no speed difference on coarse-grained, O(n) methods. I like its clarity and direct correspondence to the pure Python api for sets. Because the methods in question would all be METH_O or METH_NOARGS, the only static type checking you've lost is verifying the number of arguments. I would suggest that if you have the wrong number of arguments for s.update(t) or s.clear(), then you have problems a C API can't solve ;-) Cheers, Raymond P.S. One other thought: I don't want to crystalize the API in a way that precludes future development of the module. One possibility for the future is for updates to take multiple arguments such as s.update(t,u,v) causing three updates to be folded-in at once.

On Tue, 2006-03-21 at 22:01 -0500, Raymond Hettinger wrote:

[Me]

...

There is a semantic difference between code like s+=t and s.update(t). The former only works when t is a set and the latter works for any iterable. When the C code corresponds to the Python code, that knowledge is kept intact and there is no confusion between PyNumber_InPlaceAdd(s,t) vs PyObject_CallMethod(s, "update", "(O)", t).

Of course, that should have been s|=t and PyNumber_InPlaceOr().

Heh, my point exactly. You wouldn't have gotten confused about PySet_Update(). :) -Barry

[Barry]

Maybe it will help you to understand why I want a richer concrete API. I work on an app that is deeply integrated with Python. It's hard to say whether we embed or extend -- it's a lot of both. We use Python data structures such as lists, dicts, and sets in many places as our fundamental tracking objects.

In such an app, it would be trival to write a header: #define BarrySet_Clear(s) PyObject_CallMethod(s, "clear", NULL) Still, PyObject_Clear(s) would be better. Better still would be to examine the actual uses in the app. I suspect that most code that clears a set and then rebuilds it would be better-off starting with a new empty set (and because of freelisting, that is a very fast operation). Likewise, it only takes a one-line header to define BarrySet_Update(s). I do not want that part of the C API exposed yet. It is still under development and may eventually become a function with a variable length argument list. It's bogus to say there is some app critical need. Afterall, these are both one-line defines if you personally crave them. There's no speed argument here either -- saving an O(1) dispatch step in an O(n) operation.

there's no questions that snaking through the abstract API causes us real debugging pain

I honestly don't follow you here. Doesn't your debugger have options for step-over and step-into? Are you debugging the set module or your client code? Besides, these are all high volume functions -- do you really want to trace through the internal mechanics of set_clear? Internally, this code has special cases for small and large table sizes, it does a pointer swap with an empty table to avoid mid-stream resize issues, it treats dummy entries and active entries as being the same, and it's not at all beautiful. Ergo, it is not something you want to be tracing through. The debugging argument is bogus.

You agree that PySet_Next() is easier to use than the iterator API. We will definitely not use the latter, and if your position stands, then we'll just have to hack Python to add it (or implement it in an auxiliary module).

If you're dead-set against using the iterator API, then maybe there is something wrong with the API. You should probably start a new thread on why you detest the iterator API and see if there are ways to improve it. Avoidance of the iterator protocol is no reason to proliferate the _Next() api across other collections. That would be a mistake. It is a bug-factory. Any operation which could potentially call back arbitrary Python code can also potentially trigger a resize or table update, leaving an invalid pointer. Something as simple as PyObject_Hash(k) can trigger a callback. Usually with code like this, it would take Armin less than five minutes to write a pure Python crasher. If you absolutely must go against my recommendation, can we compromise with a semi-private _PySet_Next() so that you have a hook but without mucking-up the public API for the rest of the world?

You talk about duck typing, but I don't care about that here.

It's one of the virtues of Python that gets reflected in the abstract API. IMO, it's nice that PyObject_Dir(o) corresponds to "dir(o)" and the same for hash(o), repr(o), etc. I just hope that by hardwiring data types in stone, that your app doesn't become rigid and impossible to change. I certainly do not recommend that other people adopt this coding style (avoidance of iterators, duplication of abstact api functions in concrete form, etc.) If you're experiencing debugging pain, it may be that avoidance of abstraction is the root cause.

...

I would be supportive if you proposed a PyObject_Clear(o) function (for calling tp_clear slots when they exist and returning an error code when they don't).

I wouldn't object to that, but it wouldn't change my mind about PySet_Clear().

This is plain evidence that something is wrong with your approach. While possibly necessary in your environment, the rest of mankind should not have to stomach this kind of API clutter.

On Mar 25, 2006, at 9:57 PM, Aahz wrote:

I'd really like to see someone else who understands the issues (i.e. using the Python C-API) weigh in. Both Barry and Raymond are clever programmers who generally understand what's Pythonic, and I find myself agreeing with whoever posted last. ;-) Having another perspective would probably shed some light here.

My general preference is rather well-known, and I quote the advice I gave in "Python in a Nutshell"...: """ Some of the functions callable on specifically-typed objects [...] duplicate functionality that is also available from PyObject_ functions; in these cases, you should almost invariably use the more general PyObject_ function instead. I don’t cover such almost- redundant functions in this book. """ However, I don't go as far as suggesting PyObject_CallMethod and the like... I'd much rather have abstract-layer PyObject_... functions, as long as they're applicable to two or more concrete built-in types (for example, IMHO adding PyObject_Clear is a no-brainer -- it's obviously right). And I'm on the fence regarding the specific issue of PySet_Next. So, having carefully staked out a position smack in the middle, I cheerfully now expect to be fired upon from both sides!-) Alex

On Sun, Mar 26, 2006, Raymond Hettinger wrote:

[Alex]

...

And I'm on the fence regarding the specific issue of PySet_Next.

So, having carefully staked out a position smack in the middle, I cheerfully now expect to be fired upon from both sides!-)

Okay, here's the first cheap shot ;-) Which of the following pieces of code is preferable? The first loops with the iterator protocol and the second loops with the _next protocol.

Speaking as a person who does relatively little C programming, I don't see much difference between them. The first example is more Pythonic -- for Python. I agree with Barry that it's not much of a virtue for C code. However, I do have one nitpick with both your examples; I don't know whether this is an artifact of them being examples:

hash ^= h * 3644798167;

Seems to me that magic numbers like this need to be made constants and explained with a comment. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet

On Mar 26, 2006, at 8:43 AM, Raymond Hettinger wrote:

[Aahz]

...

Speaking as a person who does relatively little C programming, I don't see much difference between them. The first example is more Pythonic -- for Python. I agree with Barry that it's not much of a virtue for C code.

It was a trick question. Everyone is supposed to be attracted to the _next version because it is shorter, faster, and takes less ref counting management. However, the _next version has a hard-to-find bug. The call to PyObject_Hash() can trigger arbitrary Python code and possibly mutate the table, leaving pointers to invalid memory addresses. It would likely take Armin less than five minutes to write a pure Python crasher for the code. And THAT is why PySet_Next() should never come into being.

Sure, accidentally mutating underlying iterables is a subtle (but alas frequent) bug, but I don't see why it should be any harsher when the loop is using a hypothetical PySet_Next than when it is using PyIter_Next -- whatever precautions the latter takes to detect the bug and raise an exception instead of crashing, wouldn't it be at least as feasible for PySet_Next to take similar precautions (probably easier, since PySet_Next need only worry about one concrete case rather than an arbitrary variety)? What does PyDict_Next do in similar cases, and why couldn't PySet_Next behave similarly? (Yes, I could/should look it up myself, but I'm supposed to be working on the 2nd Ed of the Nutshell, whose deadline is getting worryingly close...;-). Alex

Raymond Hettinger wrote:

The difference is that the PySet_Next returns pointers to the table keys and that the mutation occurs AFTER the call to PySet_Next, leaving pointers to invalid addresses. IOW, the function cannot detect the mutation.

I'm coming late to the discussion: where did anybody ever suggest that PySet_Next should return a pointer into the set? Looking over the entire discussion, I could not find any mentioning of a specific API. If it is similar to PyDict_Next, it will have PyObject** /input/ variables, which are really meant as PyObject* /output/ variables. But yes, PyDict_Next returns a borrowed reference, so if the dictionary mutates between calls, your borrowed reference might become stale.

PyIter_Next on the other hand returns an object (not a pointer to an object such as those in the hash table).

PyIter_Next behaves identical wrt. to result types to PyDict_Next. The difference is that PyIter_Next always returns a new reference (or NULL in case of an exception). For the caller, a clear usage strategy follows from this: either discard the references before making a potentially-mutating call, or Py_INCREF the set element before making that mutating call. Of course, *after* you made the mutating call, your iteration position might be bogus, as the set might have been reorganized. If the position is represented as a Py_ssize_t (as it is for PyDict_Next), the only consequence of continuing the iteration is that you might see elements twice or not at all - you cannot cause a crash with that. Regards, Martin

Raymond Hettinger wrote:

Pardon, I bungled the terminology. PySet_Next returns a borrowed reference. That is problematic is arbitrary Python code can be run afterwards (such as PyObject_Hash in the example).

Not really. It is under the control of the caller of PySet_Next what (if any) Python code is invoked, and getting this correct is straight-forward (once you know that it yields borrowed references). I don't know what specific application Barry has in mind, but I'm sure he can get it right (although it might be an interesting experiment to test that theory :-) In general, I would expect that people find it easier to get code involving PyDict_Next right than code dealing with iterators - primarily because of the error cases you have to consider.

We have a perfectly good way to iterate with PyIter_Next(). It may take a couple of extra lines, but it is easy to get correct and has no surprises. It seems that the only issue is that Barry says that he refuses to use the iterator protocol. Heck, just turn it into a list and index directly. There is no need to muck-up the set api for this.

I don't care that much either way, although I would prefer to see an actual, current use case for PySet_Next, rather than theoretical, made-up examples. I don't expect to use Python sets in C code at all, personally. Regards, Martin

On Sun, 2006-03-26 at 13:24 -0500, Raymond Hettinger wrote:

We have a perfectly good way to iterate with PyIter_Next(). It may take a couple of extra lines, but it is easy to get correct and has no surprises. It seems that the only issue is that Barry says that he refuses to use the iterator protocol. Heck, just turn it into a list and index directly. There is no need to muck-up the set api for this.

I just think you have a narrow vision of how Python sets can be used in a C application. -Barry

On Sun, 2006-03-26 at 11:43 -0500, Raymond Hettinger wrote:

It was a trick question. Everyone is supposed to be attracted to the _next version because it is shorter, faster, and takes less ref counting management. However, the _next version has a hard-to-find bug. The call to PyObject_Hash() can trigger arbitrary Python code and possibly mutate the table, leaving pointers to invalid memory addresses. It would likely take Armin less than five minutes to write a pure Python crasher for the code. And THAT is why PySet_Next() should never come into being.

We're clearly going in circles here, and it's obvious we're not going to agree. The fact that PySet_Next() can be used incorrectly is no reason not to include it. There are /lots/ of things in Python that if you use incorrectly will screw you. So you document them, teach people when not to use them, and teach them how to use them correctly when they /are/ the right thing to use. I don't want to be babied into using inappropriate and cumbersome APIs which, yes, can be a source of their own subtle bugs. -Barry

On Mar 27, 2006, at 7:20 AM, Raymond Hettinger wrote:

Why don't we expose _PySet_Next() for Barry and leave it out of the public API for everyone else.

There are precedents for adding some functionality to the C API but not documenting it to ensure "non advanced users" don't get hurt -- that's how we added the ability to raise exceptions in different threads, in particular. Not sure if this is the best solution here, but I'm just pointing out that it's definitely not "unthinkable", procedurally speaking. Alex

Raymond Hettinger wrote:

Why don't we expose _PySet_Next() for Barry and leave it out of the public API for everyone else.

That is stupid. If Barry wants a "private" PySet_Next function, he can just implement it himself, no need to include it in the release. It should be included only if it is meant to be public. Regards, Martin

On Mon, 2006-03-27 at 10:20 -0500, Raymond Hettinger wrote:

Why don't we expose _PySet_Next() for Barry and leave it out of the public API for everyone else.

Just so I understand exactly what you mean by "leave it out of the public API", let me ask: are you saying you don't want to document the function? Do you not want to include it in setobject.h? No tests for it? If all you're suggesting is to stick an underscore in front of the name to indicate "it's something special", I could live with that, although I don't see much benefit. -Barry

We're clearly going in circles here, and it's obvious we're not going to agree.

The fact that PySet_Next() can be used incorrectly is no reason not to include it. [etc]

For what it's worth[1], I think Raymond is absolutely on crack here. [1] Not necessarily very much. There is none of my code in Python, so far as I know. * Simple API: The complexity of an API is not determined by the number of methods in it but by the variety of different things you can ask it to do, and it's not any simpler to have PyObject_CallMethod(x, "foo") PyObject_CallMethod(x, "bar") PyObject_CallMethod(x, "baz") than to have PyObject_foo(x) PyObject_bar(x) PyObject_baz(x) API complexity is measured in brain cells, not in methods. * Ease of making mistakes: The Python API is absolutely stuffed with places where you can go wrong by forgetting about subtle refcounting issues. Sure, it's nice to minimize that pain, but it's never going to be possible to write much code that uses the C API without being alert to such issues. (Incidentally, the more things you have that can only be done by invoking PyObject_CallMethod, the more places you have where you have to assume that arbitrary Python code may have been called and that reference counts may have changed behind your back.) * Duck typing: Yup, supporting duck typing is good. That's why we have an abstract API. There are concrete APIs for all sorts of particular kinds of Python object; it seems pretty clear to me that this isn't a mistake, and that sets should be one such type. Clients get to choose how to trade off the benefits in efficiency, conciseness and clarity from using the concrete API against the benefits in generality from using the abstract one. And when PySet_Add is the obvious way to add items to sets, how much C code using sets is likely to work with things that merely walk and quack like sets, anyway? * Efficiency: Anyone measured this? The mere fact that the overhead of (say) emptying a set using PyObject_CallMethod is O(1) doesn't mean it's insignificant. For many applications the size of your sets is O(1) too. (Often with quite a small implicit constant, too.) -- Gareth McCaughan

Gareth McCaughan wrote:

For what it's worth[1], I think Raymond is absolutely on crack here.

+1 on a good concrete set API from me, too. Being such similar types, sets should have about the same API richness as dicts, IMO. -- Greg

Terry Reedy wrote: [me:]

...

...

For what it's worth[1], I think Raymond is absolutely on crack here.

[Greg Ewing:]

...

+1 on a good concrete set API from me, too.

[Terry:]

For what it's worth, I think Gareth's crack at Raymond is childish and out of place here.

Er, it wasn't a crack at Raymond, it was a crack at a particular position he's taking on a particular issue. What I intended (but may have failed) to convey was: "Raymond's a clever and sensible chap, and this is a very weird position for a clever and sensible person to be taking: must be the drugs." And, just in case it's still not clear, I wasn't in fact suggesting that Raymond *is* on drugs either. However: if Raymond, or anyone else, is offended, then I'm sorry. Now, what about the technical issues, as opposed to the way I happened to introduce my comments? -- g

On Wed, 2006-03-29 at 16:29 -0500, Raymond Hettinger wrote:

The story is different for PySet_Update(). Defining it now could get in the way of possible future development for the module (the function may end-up taking a variable length argument list instead of a single argument).

So why not just go ahead and do that now? If you know that's what you want eventually, why wait? From my perspective, adding a NULL at the end of the argument list wouldn't be that big of a deal. -Barry

Barry Warsaw wrote:

On Wed, 2006-03-29 at 16:29 -0500, Raymond Hettinger wrote:

...

The story is different for PySet_Update(). Defining it now could get in the way of possible future development for the module (the function may end-up taking a variable length argument list instead of a single argument).

Would that really buy you anything much over just making multiple PySet_Update() calls? Is it just syntactic sugar, or is there some optimisation you can do with multiple updates presented all at once? -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

Raymond Hettinger wrote:

Yes, _PySet_Next() is a good compromise for you and me -- it saves you from writing a hack and saves my API from including a bug factory. The only issue is that Martin thinks it to be a crummy idea.

If it makes everyone happy, I shouldn't be in the way. Of course, it might be that not only Barry will use it, but other people as well. Regards, Martin

...

The idea is not yet ready for prime-time. If I do it for one of the named operations, I will do it for all (to keep the interface uniform).

Which named operations are you thinking of?

s.union(t), s.intersection(t), s.difference(t), s.symmetric_difference(t), s.update(t), s.intersection_update(t), s.difference_update(t), s.symmetric_difference_update(t)

if you can't make vararg PySet_Update() insanely fast, does that mean you won't add a vararg version?

Right. Please leave this one alone. I still need to work on this part of the API and do not currently have the spare clock cycles to do it right now. You don't HAVE to jam something in right away. Please let it continue to cook and not muck it up through over-enthusiasm. If I had time to explain/debate every darned aspect of what is under consideration, then it would have been done already. The fierce insistence for the patch is pre-mature and is grotesquely out-of-proportion to any potential benefit. Please do not jam this one down my throat -- the function is not necessary to have right away -- you're talking about nanoseconds of efficiency and a microscopically shorter call. Sorry, I need to stop wasting time on this thread. It has consumed far too much development time already. Please write a one-line macro for yourself and leave this alone so I can continue the development efforts at a thoughtful pace. Raymond

On Sat, 2006-03-25 at 22:05 -0500, Raymond Hettinger wrote:

Still, PyObject_Clear(s) would be better.

Although not ideal, in the interest of compromise, I could support this option. There's a problem with this though: I don't think you want to be able to clear a frozen set. My PySet_Clear() raises a SystemError and returns -1 when the object is a frozen set. If PyObject_Clear() is implemented something like int PyObject_Clear(PyObject *o) { return (o->ob_type->tp_clear ? o->ob_type->tp_clear(o) : -1); } then you /would/ be able to clear a frozen set. For that matter, it would be the case that any immutable collection would be clearable if it had a tp_clear (which it probably would). That isn't the semantics I'd expect though. That may not be solvable unless you make PyObject_Clear() an alias for PyObject_CallMethod("clear"). Although I'm sure you'll disagree, I think this is less than ideal. For one thing, you're requiring objects that work with PyObject_Clear() to implement an exact Python-level protocol (it must have a method, it must be called "clear" and it must take zero arguments). You also have to implement PyObject_Clear() with a hasattr test, because I don't think you want PyObject_Clear() raising AttributeErrors. That raises the constant overhead cost, which can make clearing small sets more expensive.

Better still would be to examine the actual uses in the app. I suspect that most code that clears a set and then rebuilds it would be better-off starting with a new empty set (and because of freelisting, that is a very fast operation).

That may not be possible. Imagine a complex application where the set is passed through many layers of calls. The set hangs off of other application level objects which you don't have access to at the point where you're deciding whether to clear the set or not. You can't create a new set because you have no way to pass the new set back to the several application level objects that would need to get their pointers updated. So the most obvious, simple approach is to just clear the set you have right there.

Likewise, it only takes a one-line header to define BarrySet_Update(s). I do not want that part of the C API exposed yet. It is still under development and may eventually become a function with a variable length argument list.

Really? That would be odd and not at all parallel with established convention (e.g. PyDict_Update()). I would think that a vararg update should be named something different in order to preserve the principle of least surprise.

If you're dead-set against using the iterator API, then maybe there is something wrong with the API. You should probably start a new thread on why you detest the iterator API and see if there are ways to improve it.

I'm not saying there's anything wrong with the iterator API, I'm saying that it's not always appropriate. It's the nail/hammer argument. But I ran out of clever when I tried to propose the simplest, most direct fix for our most pressing issues, so I'm not going to take the bait.

...

You talk about duck typing, but I don't care about that here.

It's one of the virtues of Python that gets reflected in the abstract API. IMO, it's nice that PyObject_Dir(o) corresponds to "dir(o)" and the same for hash(o), repr(o), etc. I just hope that by hardwiring data types in stone, that your app doesn't become rigid and impossible to change. I certainly do not recommend that other people adopt this coding style (avoidance of iterators, duplication of abstact api functions in concrete form, etc.) If you're experiencing debugging pain, it may be that avoidance of abstraction is the root cause.

Trust me Raymond, it's not the cause. I keep trying to explain this but I must be completely inept because you're just not getting it. Let me try this way: we're using Python's collection types (sets, lists, dicts) as our fundamental collection data structures internally in our application. There's no duck typing going on. There's no need for abstraction because we know exactly what we have and there's no chance we'll have something that smells like a set that isn't exactly a PySet. As I've said many times, I'm all for an abstract API because it's darn useful in many applications. It's the lack of a concrete API that is limiting.

...

I wouldn't object to that, but it wouldn't change my mind about PySet_Clear().

This is plain evidence that something is wrong with your approach. While possibly necessary in your environment, the rest of mankind should not have to stomach this kind of API clutter.

Please, that's a bit extreme. I haven't heard anybody scream about the PyDict's API clutter and I don't see my PySet proposal as being any different. -Barry

Greg Ewing wrote:

Hmmm, the problem here, I think, is that tp_clear is really only designed for use by the garbage collector. Giving anything else access to it is probably wrong.

Clearability is not a general feature in Python land -- a few types have a clear() method, but this is an ad hoc feature of the type concerned. I don't think it makes sense to have a general PyObject_Clear function at all.

I agree. Barry's PySet_Clear and Raymond's PyObject_Clear would be two completely unrelated functions (one invoking the "clear" method, the other invoking tp_clear). Regards, Martin

Barry, go ahead with PySet_Clear(). Raymond