Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44)
Hi Joe, all On 10/04/2008, Joe Harrington <jh@physics.ucf.edu> wrote:
Absolutely. Let's please standardize on: import numpy as np import scipy as sp
I hope we do NOT standardize on these abbreviations. While a few may have discussed it at a sprint, it hasn't seen broad discussion and there are reasons to prefer the other practice (numpy as N, scipy as S, pylab as P).
"N" is a very unfortunate choice of abbreviation, given that so many algorithms use it to indicate the number of elements in things. "np" is much safer and, like Jarrod mentioned, also only takes two keys to type. Sebastian, a simple regexp replace should fix your problem (investment in hundreds of lines of N.* usage).
My reasons for saying this go back to my reasons for disliking lots of heirarchical namespaces at all: if we must have namespaces, let's minimize the visual and typing impact by making them short and visually distinct from the function names (by capitalizing them).
The Python Style Guide (PEP08) recommends that we stick to lowercase, underscore-separated names. We'd do our users a real disservice by not following community defined standards. Namespaces throttle the amount of information with which the user is presented, and well thought through design leads to logical, intuitive segmentation of functionality. Searchable documentation and indices then become essential in guiding the user to the right place. On the other hand, when I was a freshman, we had a course on MATLAB; I remember spending countless hours using "lookfor" (I think that's what it is called?). That was one of the effects of a flat namespace. For interactive work, a flat namespace may be ideal (and I have no problem with us providing that as well), but otherwise, for file based code, I'd much prefer to have a (relatively shallow) namespace structure.
What concerns me about the discussion is that we are still not thinking like communications and thought-process experts, we are thinking like categorizers and accountants. The arguments we are raising don't have to do, positively or negatively, with the difficult acts of communicating with a computer and with other readers of our code. Those are the sole purposes of computer languages.
Isn't it easier to explain how to use a well-structured, organised library, rather than some functions-all-over-the-floor mess? If an accountant can import numpy.finance and do his work, how is that more difficult than importing every possible function included, and then sifting through them?
Namespaces add characters to code that have a high redundancy factor. This means they pollute code, make it slow and inaccurate to read, and making learning harder. Lines get longer and may wrap if they contain several calls. It is harder while visually scanning code to distinguish the function name if it's adjacent to a bunch of other text, particularly if that text appears commonly in the nearby code.
Python provides very good machinery with dealing with this verbosity: import very_foo_module as f from very.deeply.nested.namespace import func and even def foo(args): c = commonly_used_func result = c(3) + c(4) + 2*c(5) At the moment, everyone warns against using '*' with numpy, but with proper namespace, the * can be quite handy: from numpy.math import * a = sin(theta) + 3*cos(theta**2) (the example above already works in current numpy)
It therefore becomes harder to spot bugs. Mathematical code becomes less and less like the math expressions we write on paper when doing derivations, making it harder to interpret and verify. You have to memorize which subpackage each function is in, which is hard to do for those functions that could naturally go in two subpackages.
If you have need to use a subset of functions defined over different namespaces, it is very easy to create a custom module, say my_field_of_study.py: from numpy.math import cosh, sinh from numpy.linalg import inv etc. Then, a simple "from my_field_of_study import *" provides you with everything you need. This needs to be done once in your life, and can be advocated as a Cookbook recipe. Memorisation be gone (but who needs to memorise with TAB-completion anyway).
While many math function names are obvious, subpackage names are not. Is it .stat or .stats or .statistics? .rand or .random? .fin or .financial? Some functions have this problem, but *every* namespace name has it in spades.
Introspection is such a joy with IPython, or with the SAGE notebook, and many editors even provide similar functionality. Stuffing domain-specific functions into a flat namespace sounds like the ideal way of confusion a new user.
There is simply no reduction in readability, writeability, or debugability if you don't have namespace prefixes on everything, and knowing you know everything is easily accomplished now with the online categorized function list.
You're right, we read code much more often than we write it. Even so, we seldom read thousands of lines of code, and often focus on a narrow part of the module, where the namespace provides context about the origins of the function being used. The penalty on writability isn't severe, in my experience. As for debugging, I'm reminded of the speed issues with numpy.sum and python's math.sum.
We can incorporate that functionality into the doc reading apparatus ("help", currently) by using keywords in ReST comments in the docstrings and providing a way for "help" and its friends to list the keywords and what functions are connected to them.
Good idea.
What nobody has said is "if we have lots of namespaces, my code will look prettier" or "if we have lots of namespaces, normal people will learn faster" or "if we have lots of namespaces, my code will be easier to verify and debug". I don't believe any of these statements to be true. Do you?
Given that "lots" mean "the necessary amount", I'd answer "yes" to 1, 2 and 3. 1) More organised, 2) therefore easier to fit into my mind and 3) therefore easier to verify and debug.
Similarly, nobody has said, "if we have lots of namespaces, I'll be a faster coder".
We're not pushing for lots of namespaces, but for a fitting number (probably very few in the case of numpy). The impact on coding speed should be minimal.
There is a *very* high obnoxiousness factor in typing redundant stuff at an interpreter.
For the interpreted I'd suggest a different structure (i.e. an IPython profile, importing my_field_of_study.py :).
users *hate* that you have to type "print, " in order to inspect the contents of a variable.
Again, IPython -- we have the right tool for the job.
The reasons we all like Python relate to how quick and easy it is to emit code from our fingertips that is similar to what we are thinking in our brains, compared to other languages.
And your brain has a very small cache, so use it with care.
documentation does not fix. I'd rather flatten scipy, however, because the main reason to have namespaces is still satisfied that way.
I shudder to think of a flattened SciPy.
What worries me is that the EXPERIENCE of reading and writing code in Python is not much being raised in this discussion, when it should be the *key* topic of any argument about the direction of the language.
Agreed. Fortunately, we already have many talented developers working on tools to make that part easier.
So, in closing, I'd like to exhort everyone to try harder to think like a sociologist, psychologist, and linguist in addition to thinking like a computer scientist, physicist, or mathematician.
You're asking a lot -- and I don't know if that is fair. Surely, we should expect users of numpy to think a little bit like computer scientists, physicists, engineers and methematicians as well? Regards Stéfan
On Thu, Apr 10, 2008 at 6:55 AM, Stéfan van der Walt <stefan@sun.ac.za> wrote:
Hi Joe, all
On 10/04/2008, Joe Harrington <jh@physics.ucf.edu> wrote:
Absolutely. Let's please standardize on: import numpy as np import scipy as sp
I hope we do NOT standardize on these abbreviations. While a few may have discussed it at a sprint, it hasn't seen broad discussion and there are reasons to prefer the other practice (numpy as N, scipy as S, pylab as P).
"N" is a very unfortunate choice of abbreviation, given that so many algorithms use it to indicate the number of elements in things. "np" is much safer and, like Jarrod mentioned, also only takes two keys to type. Sebastian, a simple regexp replace should fix your problem (investment in hundreds of lines of N.* usage).
Hey! I use np *all the time* as an abbreviation for "number of points". I don't really see what the problem is with using numpy.whatever in library code and published scripts and whatever you want in one-off throw-away scripts. It's easy to setup a shortcut key in almost any editor to alleviate the typing burden, if that is the main objection. If you have a section of an algorithm that you are trying to make look as much like text-book pseudocode as possible, than you can't do better than "from numpy import whatever" both for clarity and python coding convention. You can also say "d = numpy.dot" in the local scope at the top of your algorithm so you can write "d(x,y)" in the algorithm itself for very pithy code that doesn't require a FAQ to understand.
Alexander Michael wrote:
On Thu, Apr 10, 2008 at 6:55 AM, Stéfan van der Walt <stefan@sun.ac.za> wrote:
Hi Joe, all
On 10/04/2008, Joe Harrington <jh@physics.ucf.edu> wrote:
Absolutely. Let's please standardize on: import numpy as np import scipy as sp
I hope we do NOT standardize on these abbreviations. While a few may have discussed it at a sprint, it hasn't seen broad discussion and there are reasons to prefer the other practice (numpy as N, scipy as S, pylab as P).
"N" is a very unfortunate choice of abbreviation, given that so many algorithms use it to indicate the number of elements in things. "np" is much safer and, like Jarrod mentioned, also only takes two keys to type. Sebastian, a simple regexp replace should fix your problem (investment in hundreds of lines of N.* usage).
Hey! I use np *all the time* as an abbreviation for "number of points". I don't really see what the problem is with using numpy.whatever in library code and published scripts and whatever you want in one-off throw-away scripts. It's easy to setup a shortcut key in almost any editor to alleviate the typing burden, if that is the main objection. If you have a section of an algorithm that you are trying to make look as much like text-book pseudocode as possible, than you can't do better than "from numpy import whatever" both for clarity and python coding convention. You can also say "d = numpy.dot" in the local scope at the top of your algorithm so you can write "d(x,y)" in the algorithm itself for very pithy code that doesn't require a FAQ to understand. _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Hi, I would prefer that the user has the choice and we have to remember that Python is dynamic typed. It is one thing to address experienced users but it takes time to get experienced. Also, some people may be just using some other code without knowing numpy is being used. That means users can 'import numpy as np' either directly or indirectly and somewhere further in their code entry 'np=1'. Python will be quite happy until the code wants a numpy function - a harsh but important lesson to learn. Readability (and debugging) is another situation where numpy is more informative than np (which is better than N) especially if someone is not familiar with numpy. Regards Bruce
On 10/04/2008, Alexander Michael <lxander.m@gmail.com> wrote:
On Thu, Apr 10, 2008 at 6:55 AM, Stéfan van der Walt <stefan@sun.ac.za> wrote:
Hi Joe, all
On 10/04/2008, Joe Harrington <jh@physics.ucf.edu> wrote:
Absolutely. Let's please standardize on: import numpy as np import scipy as sp
I hope we do NOT standardize on these abbreviations. While a few may have discussed it at a sprint, it hasn't seen broad discussion and there are reasons to prefer the other practice (numpy as N, scipy as S, pylab as P).
"N" is a very unfortunate choice of abbreviation, given that so many algorithms use it to indicate the number of elements in things. "np" is much safer and, like Jarrod mentioned, also only takes two keys to type. Sebastian, a simple regexp replace should fix your problem (investment in hundreds of lines of N.* usage).
Hey! I use np *all the time* as an abbreviation for "number of points".
Is that the de-facto abbreviation used for number of pts in your field (i.e. in literature)? If so, I'd recommend you stick to N :) Cheers Stéfan
--- Alexander Michael <lxander.m@gmail.com> wrote:
Hey! I use np *all the time* as an abbreviation for "number of points". I don't really see what the problem is with using numpy.whatever in library code and published scripts and whatever you want in one-off throw-away scripts. It's easy to setup a shortcut key in almost any editor to alleviate the typing burden, if that is the main objection. If you have a section of an algorithm that you are trying to make look as much like text-book pseudocode as possible, than you can't do better than "from numpy import whatever" both for clarity and python coding convention. You can also say "d = numpy.dot" in the local scope at the top of your algorithm so you can write "d(x,y)" in the algorithm itself for very pithy code that doesn't require a FAQ to understand.
Yes, I use np= number of points, too. But you all might want to use something else. That's the point of the flexibility of import ... as Trying to lock in namespaces as np or N or whatever is a BAD idea. Allow the flexibility. You can admonish against from ... import * for newbies and then tell them to use from ... import actual function names (as mentioned above). But locking people into a standard, even an informal one is, as someone else said, acting a bit too much like accountants. Stop, please! -- Lou Pecora, my views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Lou, On 10/04/2008, Lou Pecora <lou_boog2000@yahoo.com> wrote:
But locking people into a standard, even an informal one is, as someone else said, acting a bit too much like accountants. Stop, please!
This is just the recommended standard for code inside of numpy and scipy. You are, of course, free to do in your own code whatever you want. Regards Stéfan
Hello, I guess this maybe "normal" but it breaks a lot of thing when conterting from Numeric
a=numpy.ones(5,dtype=numpy.float32) isinstance(a[0],float) False
float64 works... I can see why one could argue for returning False, but then the converter might be too zealous things that used to work like: if type(item) in [types.IntType, types.FloatType]: or: isinstance(item, types.FloatType) now fail, should we be concerned? should we consider returning True ? Thanks, C.
isinstance() can take a tuple of classes/types as its second argument: In [1]: isinstance? Type: builtin_function_or_method Base Class: <type 'builtin_function_or_method'> String Form: <built-in function isinstance> Namespace: Python builtin Docstring: isinstance(object, class-or-type-or-tuple) -> bool Return whether an object is an instance of a class or of a subclass thereof. With a type as second argument, return whether that is the object's type. The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for isinstance(x, A) or isinstance(x, B) or ... (etc.). On Apr 10, 2008, at 12:38 PM, Charles Doutriaux wrote:
Hello,
I guess this maybe "normal" but it breaks a lot of thing when conterting from Numeric
a=numpy.ones(5,dtype=numpy.float32) isinstance(a[0],float) False
float64 works...
I can see why one could argue for returning False, but then the converter might be too zealous things that used to work like: if type(item) in [types.IntType, types.FloatType]:
or:
isinstance(item, types.FloatType)
now fail,
should we be concerned? should we consider returning True ?
Thanks,
C.
** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
On Thu, Apr 10, 2008 at 12:38 PM, Charles Doutriaux <doutriaux1@llnl.gov> wrote:
Hello,
I guess this maybe "normal" but it breaks a lot of thing when conterting from Numeric
a=numpy.ones(5,dtype=numpy.float32) isinstance(a[0],float) False
It looks like float in this case is a python float, not a numpy float.
float64 works...
It has the same underlying c type as the python float. Maybe it should fail?
I can see why one could argue for returning False, but then the converter might be too zealous things that used to work like: if type(item) in [types.IntType, types.FloatType]:
or:
isinstance(item, types.FloatType)
now fail,
should we be concerned? should we consider returning True ?
I think you want the isreal function, but it will also return true for complex with 0 imaginary part. Hmm... the various iswhatever functions seem to be lacking in coverage. Maybe we should fix that. Chuck
Charles R Harris wrote:
On Thu, Apr 10, 2008 at 12:38 PM, Charles Doutriaux <doutriaux1@llnl.gov <mailto:doutriaux1@llnl.gov>> wrote:
Hello,
I guess this maybe "normal" but it breaks a lot of thing when conterting from Numeric
>>> a=numpy.ones(5,dtype=numpy.float32) >>> isinstance(a[0],float) False >>>
It looks like float in this case is a python float, not a numpy float.
float64 works...
It has the same underlying c type as the python float. Maybe it should fail?
float64 does inherit from float so isinstance succeeds.
I can see why one could argue for returning False, but then the converter might be too zealous things that used to work like: if type(item) in [types.IntType, types.FloatType]:
or:
isinstance(item, types.FloatType)
now fail,
should we be concerned? should we consider returning True ?
I think you want the isreal function, but it will also return true for complex with 0 imaginary part. Hmm... the various iswhatever functions seem to be lacking in coverage. Maybe we should fix that.
We definitely could try and fix it.
On 10/04/2008, Charles R Harris <charlesr.harris@gmail.com> wrote:
I think you want the isreal function, but it will also return true for complex with 0 imaginary part. Hmm... the various iswhatever functions seem to be lacking in coverage. Maybe we should fix that.
icomplexobj is designed to solve that problem. Anne
Lou Pecora wrote:
That's the point of the flexibility of import ... as
Yes, it is.
Trying to lock in namespaces as np or N or whatever is a BAD idea. Allow the flexibility.
I don't think anyone is trying to lock anyone into anything.
But locking people into a standard, even an informal one is, as someone else said, acting a bit too much like accountants. Stop, please!
Locking in is bad, but establishing a standard is good. Particularly for newbies, it's really very, very helpful if example code in mailing list, in wikis, in the docs, etc, is all similar. Most folks learn by copying and pasting code form examples, having a proliferation different aliases for numpy is just asking for a lot of confusion. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Thu, Apr 10, 2008 at 10:17 AM, Lou Pecora <lou_boog2000@yahoo.com> wrote:
Yes, I use np= number of points, too. But you all might want to use something else. That's the point of the flexibility of import ... as
I would recommend against using np as a variable name. Variable names should be short and informative. I would much rather see something like "num_points".
Trying to lock in namespaces as np or N or whatever is a BAD idea. Allow the flexibility. You can admonish against from ... import * for newbies and then tell them to use from ... import actual function names (as mentioned above). But locking people into a standard, even an informal one is, as someone else said, acting a bit too much like accountants. Stop, please!
Coding standards and conventions in a large, collaborative codebase are essential. If you want to contribute to NumPy or SciPy you will have to conform to these conventions. In your own private code or private project do whatever you want. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/
On Thu, Apr 10, 2008 at 3:55 AM, Stéfan van der Walt <stefan@sun.ac.za> wrote:
Hi Joe, all
On 10/04/2008, Joe Harrington <jh@physics.ucf.edu> wrote:
Absolutely. Let's please standardize on: import numpy as np import scipy as sp
I hope we do NOT standardize on these abbreviations. While a few may have discussed it at a sprint, it hasn't seen broad discussion
Valid point... Travis did a wonderful job of summarizing that sprint and posting to the list. However, the N vs. np discussion was missed. http://projects.scipy.org/pipermail/scipy-dev/2007-December/008078.html
Namespaces throttle the amount of information with which the user is presented, and well thought through design leads to logical, intuitive segmentation of functionality.
Namespaces add characters to code that have a high redundancy factor. This means they pollute code, make it slow and inaccurate to read, and making learning harder. Lines get longer and may wrap if they contain several calls. It is harder while visually scanning code to distinguish the function name if it's adjacent to a bunch of other text, particularly if that text appears commonly in the nearby code.
I think namespaces are one of the crown-jewels that make python more attractive to scientists (not programmers) over Matlab. Even if they don't realize it yet. :) I think a lot of researchers would spend less time debugging their code if they were using python with namespaces instead of adding this: addpath(genpath('mydirectory')) in all of their Matlab code! Or other path manipulation. We certainly need a better discover mechanism for users to find functions however. -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/
On Friday 11 April 2008 04:57:31 am Christopher Burns wrote:
I think namespaces are one of the crown-jewels that make python more attractive to scientists (not programmers) over Matlab. Even if they don't realize it yet. :)
As a humble user who has neither the python-fu nor extension-fu to contribute to the numpy codebase, I second this. I have spent hours tracking down problems related to names coming from pylab vs from scipy. Now, my rule is very simple: all names in any source file come from one of 3 cases: 1. fully qualified (scipy.linalg.norm) 2. locally defined in the file 3. explicitly imported using "from xxx import ..." The only time I break this rule is when playing round in the ipython console. Rule 1 above is almost never used. With explicit name imports (practically never above 25 names), external dependencies are very clear and it is easy to figure out the extent of any effort to convert that module to C++ (usually for speed). An added bonus is an ability to automatically reload a dependent module when the code for it changes simply using a trivial script that scans dependencies in "from xxx import ..." lines -- very useful for Matlab-style code development when you have a system whose innards are under modification. Of course, this is only the view of a _user_ coming from Matlab-land. The combination of Python and C++ (especially boost.python) beats every other system that I have seen for engineering (telecommunications and statistical signal processing). Namespaces, forced indentation, and object orientation are the main reasons for me to use numpy/scipy/python. Perhaps professional programmers and software engineers know better, but the features listed above are a big part of the reason I crossed over from Matlab+C to Python+C++. Regards, Ravi
participants (13)
-
Alexander Michael -
Anne Archibald -
Bill Spotz -
Bruce Southey -
Charles Doutriaux -
Charles R Harris -
Christopher Barker -
Christopher Burns -
Jarrod Millman -
Lou Pecora -
Ravi -
Stéfan van der Walt -
Travis E. Oliphant