Need more comments from scientific community on python-dev
![](https://secure.gravatar.com/avatar/49df8cd4b1b6056c727778925f86147a.jpg?s=120&d=mm&r=g)
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information. There is a dearth of scientific-computing and number-crunching-aware people on python-dev. The result is that I sound like a lone voice arguing for something that nobody cares about. When, I don't think that is true. Please, please. If you want Python to grow support for the array interface (or something like it), then please speak up on python-dev. Even something as simple as I really see the need for a way to exchange data-format information between two objects sharing the buffer protocol can be helpful. You can post through the gmane newsgroup interface: gmane.comp.python.devel Find any of the posts on the PEP's I've introduced. Thanks for your help. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/39916bae984cb93b797efd2b175f59c0.jpg?s=120&d=mm&r=g)
On Tue, 31 Oct 2006, Travis Oliphant wrote:
The easiest access to this discussion for me was http://news.gmane.org/group/gmane.comp.python.devel/ I cannot add to this discussion, but I REALLY hope others will help Travis out here. (A few have.) He is fielding a lot of questions, some of which look to me to be from individuals who are ready to have fairly strong opinions without really understanding the "why" of his proposals. The good news is, there seems to be (on my naive reading) some sympathy for what Travis is trying to do. I think more motivating examples would prove helpful in swinging things. Cheers, Alan Isaac
![](https://secure.gravatar.com/avatar/dd7980fbcb8033634af99fca18f97062.jpg?s=120&d=mm&r=g)
Perhaps you could please elaborate on what you regard as "the issue(s)" and "what's at stake"? I'm still comparatively new (been using python and numpy circa 8-9 months), but so far I haven't encountered any problems I'd attribute to "Python's ... [in]ability to share ndarray-like information". (The most vexing problems I've had have all been with matplotlib, its dual-mode API, and its rather clumsy and haphazard documentation.) I'd like to be supportive, but not blindly-so. DG Travis Oliphant wrote:
-- HMRD/ORR/NOS/NOAA <http://response.restoration.noaa.gov/emergencyresponse/> ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.
I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.) I've suggested that numpy could use ctypes way to describe binary data a long time ago: http://sourceforge.net/mailarchive/message.php?msg_id=15395017 I agree that it would not work out of the box, but I don't see any major obstacles to adding the missing features. I think we should attempt to implement a ctypes approach and compare the result to the dtype object approach. If we discover any unimplementable features along the way, this will be a strong argument to abandon ctypes as a standard. The last time I looked at the issue, the only feature that ctypes were lacking for me was the itemsize information (endianness is probably lacking as well, but I did not need it.) I am not sure, however that this information belongs to the type description object rather than carried separately. Note that it is trivial to add an itemsize attribute to ctypes because each primitive ctype has a fixed itemsize and Structure can easily compute its itemsize. Doubling the number of ctypes to allow for different endianness is a more troubling proposition. In any case, I think we should seriously discuss the relative merits of the two approaches on this list and present our view to python-dev when we reach some consensus. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/39916bae984cb93b797efd2b175f59c0.jpg?s=120&d=mm&r=g)
On Tue, 31 Oct 2006, Sasha apparently wrote:
I think we should attempt to implement a ctypes approach
You are probably aware that Travis has addressed this at some length on comp.python.devel (for example http://article.gmane.org/gmane.comp.python.devel/84661 http://article.gmane.org/gmane.comp.python.devel/84670 ) but just in case ... Cheers, Alan Isaac ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. What advantage do we gain by making every instance of a data-type object *also* a Python type object? We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. I would, however, put effort into "undertstanding ctypes objects" as data-type objects. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant@ee.byu.edu> wrote:
Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter.
I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else. I am also not a big fan of record arrays. I believe that numeric data should be stored in "inverted tables," where columns of homogeneous data are stored contiguously. With this disclaimer, I will address a few issues below.
Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format.
Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes.
What advantage do we gain by making every instance of a data-type object *also* a Python type object?
I think the main advantage is that you can have instances:
c_int(42) c_int(42)
Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays:
But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort? Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach.
I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change.
Can you present some use cases that illustrate the advantage of your datatype approach best? I will try to implement them with ctypes.
I would, however, put effort into "undertstanding ctypes objects" as data-type objects.
Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. At the end of the day, it is better to have buffer protocol that describes the data using ctypes types than to have no standard type information at all. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
I already understand it's practical limitations --- type objects as data-type instances is too bulky and too restrictive. You have to "be a Python type object" in order to be a data-type object. I wish others would actually understand the difference. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
The letter codes are not data-type objects. The letter codes are only there for historical reasons (and they are in the struct and array modules too so blame Python...)
Sure. I understand this. This was my argument too when Numarray was first proposed. But, how do you memory-map a record where the data is actually stored differently? Numeric's answer is "you don't" but that is un-acceptable. This is why record-arrays were created and the whole pandora's box of data-types was opened.
Recent if 1 year is recent, and only if you worry precisely about when they were Python objects. Numeric always used a PyArray_Descr * structure (which was a PyObject_HEAD away from being a Python object) to describe data. The letter codes were just simple ways to "represent" those underlying structures (which could have been exposed to Python as "data-types" from the very beginning.
Sure, but you don't "need" an instance for every memory-layout description. If you want one, then great, ctypes gives it to you. But "requiring" data-type to be encoded in a type object is over-kill.
No, it's not that simple. We have a headache whenever we want to do something like I just did and separate out the concepts of what makes a Python Object a Python object. Now, we don't just modify a simple C-structure (PyArray_Descr *), we have to modify a "meta-type" or a altered dictionary and get that change put in to ctypes. Inheriting from Python type objects is harder. People who have apparently never tried seem to think it's not but it is.
No, that's not the point. The data-type object could easily be extended to accomodate those needs. The point is using data-types as instances of a regular Python object or data-types as instances of a "type" object.
This is missing the point. I have no doubt ctypes "could" be used on the Python side to describe a data-type. But, we need the PyArray_Descr * structure in NumPy. What are you going to replace the PyArray_Descr * structure with? That is the point.
Please understand what I meant. I meant putting effort into getting PyArray_DescrConverter to allow ctypes inputs and convert to the appropriate PyArray_Descr * structure. I already understand ctypes objects. I want the dtype() command to also understand them. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/25ef0a6698317c91220d6a1a89543df3.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant@ee.byu.edu> wrote: ...
I think I am starting to understand. Forgive me for being slow. Is it correct that you don't mind writing c_int * 10 instead of dtype(('<i4', 10)) as long as the result does not have to be a PyTypeObject at the C level? If this is the case, I would suggest to merge ctypes syntax with your implementation. This may help to make the case for the python-dev crowd. I believe very few people understand the subtle problems in inheriting from PyTypeObject and resist your proposal simply because they like c_int * 10 better than dtype(('<i4', 10)). There is no reason why ctypes should be implemented the way it is. It is not necessary that type(c_int()) is c_int. If a type with almost the same properties gets into the core, ctypes developers may see an advantage of simply inheriting from it and adding a factory __call__ method. Meanwhile, users familiar with ctypes will not need to learn yet another type specification syntax. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/49df8cd4b1b6056c727778925f86147a.jpg?s=120&d=mm&r=g)
Alexander Belopolsky wrote:
And I think I'm beginning to understand why some people are resistant to what I'm saying. I have no problem converting from ctypes types to the underlying PyArray_Descr structure. I don't necessarily want that to be the only way to describe new types but I sure don't mind people using that approach. I suspect it's the concern about "how to describe data-formats on the Python level" that have people concerned.
It would actually be "easier" for NumPy to understand ctypes syntax if something like a data-type object were. As it is, the converter to a PyArray_Descr structure will probably have to check the tp_name of the ctypes. Maybe this will help with the PEP to describe the difficulty of using ctypes constructs inside of other programs. And it may be a good idea to also have a get_ctype method or some-such on the ctypes attribute so that one could get a "ctypes description" from the NumPy data-type. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/ab7e74f2443b81e5175638d72be65e07.jpg?s=120&d=mm&r=g)
On 01/11/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes. There is one at the C level, it seems (though I, like perhaps most of the people on python-dev, have never actually tried using either). So perhaps, to persuade the python-dev folks, what is needed is a comparison of what has to be done at the C level. What would it take to rewrite numpy to use ctypes? There seems to be some problem with extending the type objects used by ctypes, but it's not very clear to me what that problem is (what the extensions are supposed to do). The argument that *some* datatypes format should become standard is much easier. A. M. Archibald ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/915f9e3213021d75d43bb4262167b067.jpg?s=120&d=mm&r=g)
On 11/2/06, A. M. Archibald <peridot.faceted@gmail.com> wrote:
I posted a message to the thread trying to prod things in that direction. I.e. can we see a simple concrete example of the complications involved in using ctypes interface code, vs the presumably much nicer numpy/data-descriptor code. I too think that would help convince people. Just saying it's more compilcated, trust me, doesn't help when most people reading the list have never had to write any sort of C extension. --bb ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/3f692386259303b90d1967ad662a9eb0.jpg?s=120&d=mm&r=g)
It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes.
I disagree. I can see a point in unifying ctypes and dtypes but in my mindset they've got two different scopes. ctypes is an interface to the c language dtype is a data description for the python language The biggest advantages python has over c in my mind (except the script vs compile) is that it's dynamic and expressive. I think basing dtype on ctypes is in that mind is simply destructive and against what I like best with python. ctypes mimics c's static behaviour: class struct_type(Structure): _fields_ = [....] ... ... which is somewhat static in nature. How do you create "structures" dynamically? How do you do things like this in ctypes: mytype=filter(lambda x: is_little_endian(x),map(lambda x: x==int32 and int64 or x, old_type)) b=['gg',[1,2,3]] mytype=dtype([{str: uint64 , list:another_type }[type(x)] for x in b]) I'm sure it's possibl but I'm also suspecting it not at all as clean and expressive (my second argument for choosing python over c). Since we creating something new in python isn't it best we do something the python way and later see if we can adapt the c structure to that rather than the other way around? If we can't adapt ctypes to this new thing i'm perfectly happy with leaving ctypes as is. ctype is doing a great job on the c interaction level but is plain ugly on the python level. On 11/1/06, Bill Baxter <wbaxter@gmail.com> wrote:
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/25ef0a6698317c91220d6a1a89543df3.jpg?s=120&d=mm&r=g)
On 11/3/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:
You can put the above in a function that takes fields as an argument, or type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b', c_long)]}) ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 11/3/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:
You can put the above in a function that takes fields as an argument, or type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b', c_long)]}) ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant@ee.byu.edu> wrote:
I think I am starting to understand. Forgive me for being slow. Is it correct that you don't mind writing c_int * 10 instead of dtype(('<i4', 10)) as long as the result does not have to be a PyTypeObject at the C level? If this is the case, I would suggest to merge ctypes syntax with your implementation. This may help to make the case for the python-dev crowd. I believe very few people understand the subtle problems in inheriting from PyTypeObject and resist your proposal simply because they like c_int * 10 better than dtype(('<i4', 10)). There is no reason why ctypes should be implemented the way it is. It is not necessary that type(c_int()) is c_int. If a type with almost the same properties gets into the core, ctypes developers may see an advantage of simply inheriting from it and adding a factory __call__ method. Meanwhile, users familiar with ctypes will not need to learn yet another type specification syntax. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
There's actually two issues here as well. 1) The extended buffer protocol 2) How to exchange data-format information through it. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/915f9e3213021d75d43bb4262167b067.jpg?s=120&d=mm&r=g)
One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc. I'm also in the camp that doesn't really understand this stuff well enough to really argue in favor of it, but would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages? If I've got the facts straight then I'd be happy to go and say that on comp.lang.python. --bb On 10/31/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/5f37aff3d274a0effbf20be82804d012.jpg?s=120&d=mm&r=g)
On 10/31/06, Bill Baxter <wbaxter@gmail.com> wrote:
This is a real problem, and a waste of developing time, because we lack a **general** and **standard** mechanism.
I think this is the actual reason of Travis's proposal, and of all other people like me supporting him.
If I've got the facts straight then I'd be happy to go and say that on comp.lang.python.
Please, go on! -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/915f9e3213021d75d43bb4262167b067.jpg?s=120&d=mm&r=g)
Grr, I wrote something up (it was wonderfully eloquent and convincing -- trust me!), but the gmane interface seems to have lost it (I posted two things before responding to the two "authorization requested" emails, and only one of the two things ever got posted it seems.) --bb On 11/2/06, Lisandro Dalcin <dalcinl@gmail.com> wrote:
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/5f37aff3d274a0effbf20be82804d012.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.
I believe there are another really important reason to support Travis's proposal. Python should have a general, simple, powerful and standard way to associate type metadata to binary streams. This enables efficient share or memory, but also can help a lot to achieve inter-language capabilities. As an example, I would to make some comments about MPI, the "de facto" standard for message-passing in parallel computing. When MPI specification was being developed, there were many different, incompatible libraries for message passing, and many of them (ie. PVM) usually communicated binary data following a pack/unpack approach. The MPI Forum decided to add an alternative approach, adding a class 'MPI_Datatype', and many predefined instances of this class (MPI_INT, MPI_FLOAT, etc.) in order to represent the basic datatypes (integers, floats, strings, complex numbers) in C, Fortran, and C++. Predefined MPI datatypes can be used as building blocks for describing very complicated memory layouts and derived,nested structures. This way, any implementations of this standard can support interprocess communication in a multi-language, muti-architecture approach. MPI datatypes can also be "decoded" in order to get the basic, predefined types contained in a user-defined datatype. I think the MPI example, despite being taken from a very specific domain with particular needs, has a strong connection with Travis's proposal. Pack/unpack degrades performance, but if you want to efficiently share binary data (with other languages/architecture), then a general and extensible mechanism for attaching type metadata to binary streams must me defined. I would like to know your opinions about this. Travis's datatypes in Python should be more that something to share data, they should be able to **describe** data. And I am not sure if ctypes can be be the way. Can Jython take advantage of ctypes??? Or in the near future CPython/Jython will have a 'jtypes' module? I forget! Travis/Pearu, why don't you develop a 'ftypes' for Fortran binary data, and next ask Python core developers to include it in standard library (a joke, of course)? Travis, if you think my previous example about MPI can help, I can resend this post to Python-Dev. I ask you this because I really want to help and not to make noise. Regars, -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/39916bae984cb93b797efd2b175f59c0.jpg?s=120&d=mm&r=g)
On Tue, 31 Oct 2006, Travis Oliphant wrote:
The easiest access to this discussion for me was http://news.gmane.org/group/gmane.comp.python.devel/ I cannot add to this discussion, but I REALLY hope others will help Travis out here. (A few have.) He is fielding a lot of questions, some of which look to me to be from individuals who are ready to have fairly strong opinions without really understanding the "why" of his proposals. The good news is, there seems to be (on my naive reading) some sympathy for what Travis is trying to do. I think more motivating examples would prove helpful in swinging things. Cheers, Alan Isaac
![](https://secure.gravatar.com/avatar/dd7980fbcb8033634af99fca18f97062.jpg?s=120&d=mm&r=g)
Perhaps you could please elaborate on what you regard as "the issue(s)" and "what's at stake"? I'm still comparatively new (been using python and numpy circa 8-9 months), but so far I haven't encountered any problems I'd attribute to "Python's ... [in]ability to share ndarray-like information". (The most vexing problems I've had have all been with matplotlib, its dual-mode API, and its rather clumsy and haphazard documentation.) I'd like to be supportive, but not blindly-so. DG Travis Oliphant wrote:
-- HMRD/ORR/NOS/NOAA <http://response.restoration.noaa.gov/emergencyresponse/> ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.
I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.) I've suggested that numpy could use ctypes way to describe binary data a long time ago: http://sourceforge.net/mailarchive/message.php?msg_id=15395017 I agree that it would not work out of the box, but I don't see any major obstacles to adding the missing features. I think we should attempt to implement a ctypes approach and compare the result to the dtype object approach. If we discover any unimplementable features along the way, this will be a strong argument to abandon ctypes as a standard. The last time I looked at the issue, the only feature that ctypes were lacking for me was the itemsize information (endianness is probably lacking as well, but I did not need it.) I am not sure, however that this information belongs to the type description object rather than carried separately. Note that it is trivial to add an itemsize attribute to ctypes because each primitive ctype has a fixed itemsize and Structure can easily compute its itemsize. Doubling the number of ctypes to allow for different endianness is a more troubling proposition. In any case, I think we should seriously discuss the relative merits of the two approaches on this list and present our view to python-dev when we reach some consensus. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/39916bae984cb93b797efd2b175f59c0.jpg?s=120&d=mm&r=g)
On Tue, 31 Oct 2006, Sasha apparently wrote:
I think we should attempt to implement a ctypes approach
You are probably aware that Travis has addressed this at some length on comp.python.devel (for example http://article.gmane.org/gmane.comp.python.devel/84661 http://article.gmane.org/gmane.comp.python.devel/84670 ) but just in case ... Cheers, Alan Isaac ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. What advantage do we gain by making every instance of a data-type object *also* a Python type object? We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. I would, however, put effort into "undertstanding ctypes objects" as data-type objects. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant@ee.byu.edu> wrote:
Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter.
I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else. I am also not a big fan of record arrays. I believe that numeric data should be stored in "inverted tables," where columns of homogeneous data are stored contiguously. With this disclaimer, I will address a few issues below.
Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format.
Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes.
What advantage do we gain by making every instance of a data-type object *also* a Python type object?
I think the main advantage is that you can have instances:
c_int(42) c_int(42)
Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays:
But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort? Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach.
I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change.
Can you present some use cases that illustrate the advantage of your datatype approach best? I will try to implement them with ctypes.
I would, however, put effort into "undertstanding ctypes objects" as data-type objects.
Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. At the end of the day, it is better to have buffer protocol that describes the data using ctypes types than to have no standard type information at all. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
I already understand it's practical limitations --- type objects as data-type instances is too bulky and too restrictive. You have to "be a Python type object" in order to be a data-type object. I wish others would actually understand the difference. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
The letter codes are not data-type objects. The letter codes are only there for historical reasons (and they are in the struct and array modules too so blame Python...)
Sure. I understand this. This was my argument too when Numarray was first proposed. But, how do you memory-map a record where the data is actually stored differently? Numeric's answer is "you don't" but that is un-acceptable. This is why record-arrays were created and the whole pandora's box of data-types was opened.
Recent if 1 year is recent, and only if you worry precisely about when they were Python objects. Numeric always used a PyArray_Descr * structure (which was a PyObject_HEAD away from being a Python object) to describe data. The letter codes were just simple ways to "represent" those underlying structures (which could have been exposed to Python as "data-types" from the very beginning.
Sure, but you don't "need" an instance for every memory-layout description. If you want one, then great, ctypes gives it to you. But "requiring" data-type to be encoded in a type object is over-kill.
No, it's not that simple. We have a headache whenever we want to do something like I just did and separate out the concepts of what makes a Python Object a Python object. Now, we don't just modify a simple C-structure (PyArray_Descr *), we have to modify a "meta-type" or a altered dictionary and get that change put in to ctypes. Inheriting from Python type objects is harder. People who have apparently never tried seem to think it's not but it is.
No, that's not the point. The data-type object could easily be extended to accomodate those needs. The point is using data-types as instances of a regular Python object or data-types as instances of a "type" object.
This is missing the point. I have no doubt ctypes "could" be used on the Python side to describe a data-type. But, we need the PyArray_Descr * structure in NumPy. What are you going to replace the PyArray_Descr * structure with? That is the point.
Please understand what I meant. I meant putting effort into getting PyArray_DescrConverter to allow ctypes inputs and convert to the appropriate PyArray_Descr * structure. I already understand ctypes objects. I want the dtype() command to also understand them. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/25ef0a6698317c91220d6a1a89543df3.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant@ee.byu.edu> wrote: ...
I think I am starting to understand. Forgive me for being slow. Is it correct that you don't mind writing c_int * 10 instead of dtype(('<i4', 10)) as long as the result does not have to be a PyTypeObject at the C level? If this is the case, I would suggest to merge ctypes syntax with your implementation. This may help to make the case for the python-dev crowd. I believe very few people understand the subtle problems in inheriting from PyTypeObject and resist your proposal simply because they like c_int * 10 better than dtype(('<i4', 10)). There is no reason why ctypes should be implemented the way it is. It is not necessary that type(c_int()) is c_int. If a type with almost the same properties gets into the core, ctypes developers may see an advantage of simply inheriting from it and adding a factory __call__ method. Meanwhile, users familiar with ctypes will not need to learn yet another type specification syntax. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/49df8cd4b1b6056c727778925f86147a.jpg?s=120&d=mm&r=g)
Alexander Belopolsky wrote:
And I think I'm beginning to understand why some people are resistant to what I'm saying. I have no problem converting from ctypes types to the underlying PyArray_Descr structure. I don't necessarily want that to be the only way to describe new types but I sure don't mind people using that approach. I suspect it's the concern about "how to describe data-formats on the Python level" that have people concerned.
It would actually be "easier" for NumPy to understand ctypes syntax if something like a data-type object were. As it is, the converter to a PyArray_Descr structure will probably have to check the tp_name of the ctypes. Maybe this will help with the PEP to describe the difficulty of using ctypes constructs inside of other programs. And it may be a good idea to also have a get_ctype method or some-such on the ctypes attribute so that one could get a "ctypes description" from the NumPy data-type. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/ab7e74f2443b81e5175638d72be65e07.jpg?s=120&d=mm&r=g)
On 01/11/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes. There is one at the C level, it seems (though I, like perhaps most of the people on python-dev, have never actually tried using either). So perhaps, to persuade the python-dev folks, what is needed is a comparison of what has to be done at the C level. What would it take to rewrite numpy to use ctypes? There seems to be some problem with extending the type objects used by ctypes, but it's not very clear to me what that problem is (what the extensions are supposed to do). The argument that *some* datatypes format should become standard is much easier. A. M. Archibald ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/915f9e3213021d75d43bb4262167b067.jpg?s=120&d=mm&r=g)
On 11/2/06, A. M. Archibald <peridot.faceted@gmail.com> wrote:
I posted a message to the thread trying to prod things in that direction. I.e. can we see a simple concrete example of the complications involved in using ctypes interface code, vs the presumably much nicer numpy/data-descriptor code. I too think that would help convince people. Just saying it's more compilcated, trust me, doesn't help when most people reading the list have never had to write any sort of C extension. --bb ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/3f692386259303b90d1967ad662a9eb0.jpg?s=120&d=mm&r=g)
It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes.
I disagree. I can see a point in unifying ctypes and dtypes but in my mindset they've got two different scopes. ctypes is an interface to the c language dtype is a data description for the python language The biggest advantages python has over c in my mind (except the script vs compile) is that it's dynamic and expressive. I think basing dtype on ctypes is in that mind is simply destructive and against what I like best with python. ctypes mimics c's static behaviour: class struct_type(Structure): _fields_ = [....] ... ... which is somewhat static in nature. How do you create "structures" dynamically? How do you do things like this in ctypes: mytype=filter(lambda x: is_little_endian(x),map(lambda x: x==int32 and int64 or x, old_type)) b=['gg',[1,2,3]] mytype=dtype([{str: uint64 , list:another_type }[type(x)] for x in b]) I'm sure it's possibl but I'm also suspecting it not at all as clean and expressive (my second argument for choosing python over c). Since we creating something new in python isn't it best we do something the python way and later see if we can adapt the c structure to that rather than the other way around? If we can't adapt ctypes to this new thing i'm perfectly happy with leaving ctypes as is. ctype is doing a great job on the c interaction level but is plain ugly on the python level. On 11/1/06, Bill Baxter <wbaxter@gmail.com> wrote:
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/25ef0a6698317c91220d6a1a89543df3.jpg?s=120&d=mm&r=g)
On 11/3/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:
You can put the above in a function that takes fields as an argument, or type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b', c_long)]}) ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 11/3/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:
You can put the above in a function that takes fields as an argument, or type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b', c_long)]}) ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/837d314801b4f1400d6eabc767ca2cac.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant@ee.byu.edu> wrote:
I think I am starting to understand. Forgive me for being slow. Is it correct that you don't mind writing c_int * 10 instead of dtype(('<i4', 10)) as long as the result does not have to be a PyTypeObject at the C level? If this is the case, I would suggest to merge ctypes syntax with your implementation. This may help to make the case for the python-dev crowd. I believe very few people understand the subtle problems in inheriting from PyTypeObject and resist your proposal simply because they like c_int * 10 better than dtype(('<i4', 10)). There is no reason why ctypes should be implemented the way it is. It is not necessary that type(c_int()) is c_int. If a type with almost the same properties gets into the core, ctypes developers may see an advantage of simply inheriting from it and adding a factory __call__ method. Meanwhile, users familiar with ctypes will not need to learn yet another type specification syntax. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/4d021a1d1319f36ad861ebef0eb5ba44.jpg?s=120&d=mm&r=g)
Sasha wrote:
There's actually two issues here as well. 1) The extended buffer protocol 2) How to exchange data-format information through it. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/915f9e3213021d75d43bb4262167b067.jpg?s=120&d=mm&r=g)
One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc. I'm also in the camp that doesn't really understand this stuff well enough to really argue in favor of it, but would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages? If I've got the facts straight then I'd be happy to go and say that on comp.lang.python. --bb On 10/31/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/5f37aff3d274a0effbf20be82804d012.jpg?s=120&d=mm&r=g)
On 10/31/06, Bill Baxter <wbaxter@gmail.com> wrote:
This is a real problem, and a waste of developing time, because we lack a **general** and **standard** mechanism.
I think this is the actual reason of Travis's proposal, and of all other people like me supporting him.
If I've got the facts straight then I'd be happy to go and say that on comp.lang.python.
Please, go on! -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
![](https://secure.gravatar.com/avatar/915f9e3213021d75d43bb4262167b067.jpg?s=120&d=mm&r=g)
Grr, I wrote something up (it was wonderfully eloquent and convincing -- trust me!), but the gmane interface seems to have lost it (I posted two things before responding to the two "authorization requested" emails, and only one of the two things ever got posted it seems.) --bb On 11/2/06, Lisandro Dalcin <dalcinl@gmail.com> wrote:
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/5f37aff3d274a0effbf20be82804d012.jpg?s=120&d=mm&r=g)
On 10/31/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information.
I believe there are another really important reason to support Travis's proposal. Python should have a general, simple, powerful and standard way to associate type metadata to binary streams. This enables efficient share or memory, but also can help a lot to achieve inter-language capabilities. As an example, I would to make some comments about MPI, the "de facto" standard for message-passing in parallel computing. When MPI specification was being developed, there were many different, incompatible libraries for message passing, and many of them (ie. PVM) usually communicated binary data following a pack/unpack approach. The MPI Forum decided to add an alternative approach, adding a class 'MPI_Datatype', and many predefined instances of this class (MPI_INT, MPI_FLOAT, etc.) in order to represent the basic datatypes (integers, floats, strings, complex numbers) in C, Fortran, and C++. Predefined MPI datatypes can be used as building blocks for describing very complicated memory layouts and derived,nested structures. This way, any implementations of this standard can support interprocess communication in a multi-language, muti-architecture approach. MPI datatypes can also be "decoded" in order to get the basic, predefined types contained in a user-defined datatype. I think the MPI example, despite being taken from a very specific domain with particular needs, has a strong connection with Travis's proposal. Pack/unpack degrades performance, but if you want to efficiently share binary data (with other languages/architecture), then a general and extensible mechanism for attaching type metadata to binary streams must me defined. I would like to know your opinions about this. Travis's datatypes in Python should be more that something to share data, they should be able to **describe** data. And I am not sure if ctypes can be be the way. Can Jython take advantage of ctypes??? Or in the near future CPython/Jython will have a 'jtypes' module? I forget! Travis/Pearu, why don't you develop a 'ftypes' for Fortran binary data, and next ask Python core developers to include it in standard library (a joke, of course)? Travis, if you think my previous example about MPI can help, I can resend this post to Python-Dev. I ask you this because I really want to help and not to make noise. Regars, -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
participants (11)
-
A. M. Archibald
-
Alan G Isaac
-
Alan Isaac
-
Alexander Belopolsky
-
Bill Baxter
-
David L Goldsmith
-
Lisandro Dalcin
-
Sasha
-
Torgil Svensson
-
Travis Oliphant
-
Travis Oliphant