On Tue, 31 Oct 2006, Travis Oliphant wrote:

Please, please. If you want Python to grow support for the array interface (or something like it), then please speak up on python-dev.

The easiest access to this discussion for me was http://news.gmane.org/group/gmane.comp.python.devel/ I cannot add to this discussion, but I REALLY hope others will help Travis out here. (A few have.) He is fielding a lot of questions, some of which look to me to be from individuals who are ready to have fairly strong opinions without really understanding the "why" of his proposals. The good news is, there seems to be (on my naive reading) some sympathy for what Travis is trying to do. I think more motivating examples would prove helpful in swinging things. Cheers, Alan Isaac

On 10/31/06, Travis Oliphant wrote:

I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.

I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.) I've suggested that numpy could use ctypes way to describe binary data a long time ago: http://sourceforge.net/mailarchive/message.php?msg_id=15395017 I agree that it would not work out of the box, but I don't see any major obstacles to adding the missing features. I think we should attempt to implement a ctypes approach and compare the result to the dtype object approach. If we discover any unimplementable features along the way, this will be a strong argument to abandon ctypes as a standard. The last time I looked at the issue, the only feature that ctypes were lacking for me was the itemsize information (endianness is probably lacking as well, but I did not need it.) I am not sure, however that this information belongs to the type description object rather than carried separately. Note that it is trivial to add an itemsize attribute to ctypes because each primitive ctype has a fixed itemsize and Structure can easily compute its itemsize. Doubling the number of ctypes to allow for different endianness is a more troubling proposition. In any case, I think we should seriously discuss the relative merits of the two approaches on this list and present our view to python-dev when we reach some consensus. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Sasha wrote:

On 10/31/06, Travis Oliphant wrote:

...

I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.

I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.)

Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. What advantage do we gain by making every instance of a data-type object *also* a Python type object? We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. I would, however, put effort into "undertstanding ctypes objects" as data-type objects. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

On 10/31/06, Travis Oliphant wrote:

Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter.

I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else. I am also not a big fan of record arrays. I believe that numeric data should be stored in "inverted tables," where columns of homogeneous data are stored contiguously. With this disclaimer, I will address a few issues below.

Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format.

Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes.

What advantage do we gain by making every instance of a data-type object *also* a Python type object?

I think the main advantage is that you can have instances:

...

...

c_int(42) c_int(42)

Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays:

...

...

a10 = c_int*10 a10() <__main__.c_int_Array_10 object at 0x2a95816958>

We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument.

But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort? Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach.

I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change.

Can you present some use cases that illustrate the advantage of your datatype approach best? I will try to implement them with ctypes.

I would, however, put effort into "undertstanding ctypes objects" as data-type objects.

Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. At the end of the day, it is better to have buffer protocol that describes the data using ctypes types than to have no standard type information at all. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Sasha wrote:

On 10/31/06, Travis Oliphant wrote:

...

Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter.

I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else.

The letter codes are not data-type objects. The letter codes are only there for historical reasons (and they are in the struct and array modules too so blame Python...)

I am also not a big fan of record arrays. I believe that numeric data should be stored in "inverted tables," where columns of homogeneous data are stored contiguously.

Sure. I understand this. This was my argument too when Numarray was first proposed. But, how do you memory-map a record where the data is actually stored differently? Numeric's answer is "you don't" but that is un-acceptable. This is why record-arrays were created and the whole pandora's box of data-types was opened.

...

Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format.

Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes.

Recent if 1 year is recent, and only if you worry precisely about when they were Python objects. Numeric always used a PyArray_Descr * structure (which was a PyObject_HEAD away from being a Python object) to describe data. The letter codes were just simple ways to "represent" those underlying structures (which could have been exposed to Python as "data-types" from the very beginning.

...

What advantage do we gain by making every instance of a data-type object *also* a Python type object?

I think the main advantage is that you can have instances:

...

...

...

c_int(42)

c_int(42)

Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays:

Sure, but you don't "need" an instance for every memory-layout description. If you want one, then great, ctypes gives it to you. But "requiring" data-type to be encoded in a type object is over-kill.

...

We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument.

But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort?

No, it's not that simple. We have a headache whenever we want to do something like I just did and separate out the concepts of what makes a Python Object a Python object. Now, we don't just modify a simple C-structure (PyArray_Descr *), we have to modify a "meta-type" or a altered dictionary and get that change put in to ctypes. Inheriting from Python type objects is harder. People who have apparently never tried seem to think it's not but it is.

Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach.

No, that's not the point. The data-type object could easily be extended to accomodate those needs. The point is using data-types as instances of a regular Python object or data-types as instances of a "type" object.

...

I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change.

Can you present some use cases that illustrate the advantage of your datatype approach best?

I will try to implement them with ctypes.

This is missing the point. I have no doubt ctypes "could" be used on the Python side to describe a data-type. But, we need the PyArray_Descr * structure in NumPy. What are you going to replace the PyArray_Descr * structure with? That is the point.

...

I would, however, put effort into "undertstanding ctypes objects" as data-type objects.

Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative.

Please understand what I meant. I meant putting effort into getting PyArray_DescrConverter to allow ctypes inputs and convert to the appropriate PyArray_Descr * structure. I already understand ctypes objects. I want the dtype() command to also understand them. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc. I'm also in the camp that doesn't really understand this stuff well enough to really argue in favor of it, but would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages? If I've got the facts straight then I'd be happy to go and say that on comp.lang.python. --bb On 10/31/06, Travis Oliphant wrote:

I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.

There is a dearth of scientific-computing and number-crunching-aware people on python-dev. The result is that I sound like a lone voice arguing for something that nobody cares about. When, I don't think that is true. Please, please. If you want Python to grow support for the array interface (or something like it), then please speak up on python-dev.

Even something as simple as I really see the need for a way to exchange data-format information between two objects sharing the buffer protocol can be helpful.

You can post through the gmane newsgroup interface:

gmane.comp.python.devel

Find any of the posts on the PEP's I've introduced. Thanks for your help.

-Travis

------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Grr, I wrote something up (it was wonderfully eloquent and convincing -- trust me!), but the gmane interface seems to have lost it (I posted two things before responding to the two "authorization requested" emails, and only one of the two things ever got posted it seems.) --bb On 11/2/06, Lisandro Dalcin wrote:

On 10/31/06, Bill Baxter wrote:

...

One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc.

This is a real problem, and a waste of developing time, because we lack a **general** and **standard** mechanism.

...

would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages?

I think this is the actual reason of Travis's proposal, and of all other people like me supporting him.

...

If I've got the facts straight then I'd be happy to go and say that on comp.lang.python.

Please, go on!

------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

On 10/31/06, Travis Oliphant wrote:

I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.

I believe there are another really important reason to support Travis's proposal. Python should have a general, simple, powerful and standard way to associate type metadata to binary streams. This enables efficient share or memory, but also can help a lot to achieve inter-language capabilities. As an example, I would to make some comments about MPI, the "de facto" standard for message-passing in parallel computing. When MPI specification was being developed, there were many different, incompatible libraries for message passing, and many of them (ie. PVM) usually communicated binary data following a pack/unpack approach. The MPI Forum decided to add an alternative approach, adding a class 'MPI_Datatype', and many predefined instances of this class (MPI_INT, MPI_FLOAT, etc.) in order to represent the basic datatypes (integers, floats, strings, complex numbers) in C, Fortran, and C++. Predefined MPI datatypes can be used as building blocks for describing very complicated memory layouts and derived,nested structures. This way, any implementations of this standard can support interprocess communication in a multi-language, muti-architecture approach. MPI datatypes can also be "decoded" in order to get the basic, predefined types contained in a user-defined datatype. I think the MPI example, despite being taken from a very specific domain with particular needs, has a strong connection with Travis's proposal. Pack/unpack degrades performance, but if you want to efficiently share binary data (with other languages/architecture), then a general and extensible mechanism for attaching type metadata to binary streams must me defined. I would like to know your opinions about this. Travis's datatypes in Python should be more that something to share data, they should be able to **describe** data. And I am not sure if ctypes can be be the way. Can Jython take advantage of ctypes??? Or in the near future CPython/Jython will have a 'jtypes' module? I forget! Travis/Pearu, why don't you develop a 'ftypes' for Fortran binary data, and next ask Python core developers to include it in standard library (a joke, of course)? Travis, if you think my previous example about MPI can help, I can resend this post to Python-Dev. I ask you this because I really want to help and not to make noise. Regars, -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

On 01/11/06, Travis Oliphant wrote:

And it may be a good idea to also have a get_ctype method or some-such on the ctypes attribute so that one could get a "ctypes description" from the NumPy data-type.

It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes. There is one at the C level, it seems (though I, like perhaps most of the people on python-dev, have never actually tried using either). So perhaps, to persuade the python-dev folks, what is needed is a comparison of what has to be done at the C level. What would it take to rewrite numpy to use ctypes? There seems to be some problem with extending the type objects used by ctypes, but it's not very clear to me what that problem is (what the extensions are supposed to do). The argument that *some* datatypes format should become standard is much easier. A. M. Archibald ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

...

It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes.

I disagree. I can see a point in unifying ctypes and dtypes but in my mindset they've got two different scopes. ctypes is an interface to the c language dtype is a data description for the python language The biggest advantages python has over c in my mind (except the script vs compile) is that it's dynamic and expressive. I think basing dtype on ctypes is in that mind is simply destructive and against what I like best with python. ctypes mimics c's static behaviour: class struct_type(Structure): _fields_ = [....] ... ... which is somewhat static in nature. How do you create "structures" dynamically? How do you do things like this in ctypes: mytype=filter(lambda x: is_little_endian(x),map(lambda x: x==int32 and int64 or x, old_type)) b=['gg',[1,2,3]] mytype=dtype([{str: uint64 , list:another_type }[type(x)] for x in b]) I'm sure it's possibl but I'm also suspecting it not at all as clean and expressive (my second argument for choosing python over c). Since we creating something new in python isn't it best we do something the python way and later see if we can adapt the c structure to that rather than the other way around? If we can't adapt ctypes to this new thing i'm perfectly happy with leaving ctypes as is. ctype is doing a great job on the c interaction level but is plain ugly on the python level. On 11/1/06, Bill Baxter wrote:

On 11/2/06, A. M. Archibald wrote:

...

On 01/11/06, Travis Oliphant wrote:

...

And it may be a good idea to also have a get_ctype method or some-such on the ctypes attribute so that one could get a "ctypes description" from the NumPy data-type.

It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes. There is one at the C level, it seems (though I, like perhaps most of the people on python-dev, have never actually tried using either). So perhaps, to persuade the python-dev folks, what is needed is a comparison of what has to be done at the C level. What would it take to rewrite numpy to use ctypes? There seems to be some problem with extending the type objects used by ctypes, but it's not very clear to me what that problem is (what the extensions are supposed to do).

I posted a message to the thread trying to prod things in that direction. I.e. can we see a simple concrete example of the complications involved in using ctypes interface code, vs the presumably much nicer numpy/data-descriptor code. I too think that would help convince people. Just saying it's more compilcated, trust me, doesn't help when most people reading the list have never had to write any sort of C extension.

--bb

------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

On 11/3/06, Torgil Svensson wrote:

class struct_type(Structure): _fields_ = [....]

... ... which is somewhat static in nature. How do you create "structures" dynamically?

You can put the above in a function that takes fields as an argument, or type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b', c_long)]}) ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642