Mailman 3 June 2009 - NumPy-Discussion

Import Numpy on Windows Vista x64 AMD
by Dinesh B Vadhia 28 Jun '09

28 Jun '09

Ticket# 1084 (http://projects.scipy.org/numpy/timeline?from=2009-06-09T03%3A01%3A59-0500&…) says that the numpy import on Windows Vista x64 AMD systems works now. Is this for Numpy 1.3 or 1.4 and if 1.3 has anyone tried it successfully? Thanks. Dinesh

2 5

switching to float32
by John Schulman 28 Jun '09

28 Jun '09

I'm trying to reduce the memory used in a calculation, so I'd like to switch my program to float32 instead of float64. Is it possible to change the numpy default float size, so I don't have to explicitly state dtype=np.float32 everywhere? Thanks, John

6 7

Fwd: YASS (Yet Another Success Story)
by Emanuel Woiski 27 Jun '09

27 Jun '09

pessoal Hstória de sucesso woiski ---------- Forwarded message ---------- From: k3xji <sumerc(a)gmail.com> Date: 2009/6/20 Subject: YASS (Yet Another Success Story) To: python-list(a)python.org Hi all, Started a project year ago with hard goals in mind : Developing a game server which can handle thousands of clients simultaneously.System must be stable, scalable, efficient, and if the client need to migrate the server to another OS, this will be a matter of time, if possible without changing any code. Also, the system should be controlled via database operations, meaning some specific DB changes should be determined by the system on-the-fly and necassary processing should be done at runtime(Client timeout values, adding new game lobbies..etc..) And all of this should be a one-man project dues to the fact that the project is a volunteer work, thus is separate from our ordinary jobs, there is no budget:). This was 8 months ago, and I still remember we are talking with my colleguae to decide which programming language is better for our requirements. We wrote the tradeoffs of different languages on board and compare our objectives against them by giving numbers. I remember, the last three standing programming languages are: C++, Java, and Python. We decided that code maintability and stability is cheaper than native code efficiency and also easy porting is important. C++ is only successfull on efficiency but maintaining the code, even if we use 3rd party libraries is a nightmare, especially in a massive multithreaded project like a game server. Also, we decided a game server's bottleneck is on network I/O other than the actual processing which means there should be not so much performance degradation between C++ and an interpreted language. So the final is between Java and Python. I know Java has been adding very good optimizations for interpreter efficiency over the years(JIT compiling..etc.) And I remember somewhere reading that Java is 10 times faster than Python. But Python, on the other hand is a perfect language and very simple which means the project may be completed sooner and efficiency again will be not a so much problem as the bottleneck should be on network I/O if the server is coded correctly. Although we have made this decision, we still had some doubts but we move on with Python. I don't know Python at the time and only coded few simple projects with it. I started with reading python socket/threading tutorials, then I downloaded and read actual Python code to get deeper and deeper. This was 10 months ago. Now we have been able to implement the game server with Python using MySQL as a backend DB and currently 5000 players (increasing) are playing multiplayer games with it with a system load of 0.4-0.8(see Unix getloadavg output) on a 1.8 Ghz Dual Xeon processor. Let me give some benchmarks, please note that one thing I learn through this journey is that testing/benchmarking a server software is really difficult and maybe in some conditions impossible thing to do correctly as inactive client connections may cause indeterministic behavior and also WAN latencies are very different than LAN latencies..etc. Anyway, we had to do a benchmark before going with this server on live. We installed a UnrealIRCd server which is also using select() as an I/O paradigm just as we are and supporting many of the features we have. This server is a IRC server written in C and being written since nearly 10 years. We connect thousands of clients and do simple messaging between them. I don't have an official test/benchmark result or a nice graph for this, however the system running UnrealIRCD is locked up when processing 4 Mb/sec of data, where our 6 month old Python server is locking up the system resources after 3.5 Mb/sec. We have done other tests including stressing server accept(), adding inactive connections but I am really not %100 confident about the test environments and do not want to give you false impression. But we decided that this 0.5 Mb factor is still not enough for us and we move on to inspect what is really causing the bottlenecks and how to optimize them. Here, to be honest, I cannot see very uch help from Python. Profiling/Optimization is the only place I am upset with Python. Here is the requirements for the profiler: I want to be able to enable/disable profiler tool on the fly without requiring any kind of restart and the profiler should be multithreaded (thus thread-safe). I decided that cProfiler and other profiler tools are in need of some updates to fulfill my reqırements, which makes me write my own profiler. It took me 3 days to come up with a profiler that satisfies my requirements, this profiler will profile only my code, meaning the server code to avoid performance degradation and timings will need not be nested-aware and timing precision is not very important (other than secs). I have implemented the profiler with the help of decorators and started profiling my code on-live. Here comes the another part of Python that is some kind of shady: optimization. After profiling the code, it turns out most of the time is spent on the following: 1) Selecting/Deselecting interest sets for select() 2) Unnecessary function calls. (for integrating some OOP concepts, I think we have overlooked the performance factor and there really are some functions which can be inlined but declared as a function) 3) Redundant try-except's in all over place(Again our fault to make the system stable, we have put some debug purposed-assert like try- excepts in the main server flow.) After, looking on these subjects for 2 weeks, we came up with solutions for all of them, and we see that it is working approximately 5 times faster. This test is done in live, we run two server processes which are all running on a predefined CPU(means CPU affinity of the process is set before hand to avoid mis-interpreting system diagnose tools like top(1)). When same client number is connected to both servers, the second one is giving 5 times better values in top and ps tools. Also, the processing time (meaning the server loop except the select() call) is also 10 times faster than before. Just one note about optimizing Python code: do not optimize Python code based on your assumptions, just go and test if it really runs faster. I don't want to go to details of this hint, but believe me making Python code optimized may be very very tricky. It is then I decided to write up here this as a success story, as I am very newcomer to Python but come up with a nearly commercial product in a very short period of time and I don't think this is about my personal characteristics and intelligence or so:), as I am old enough to know/meet that there are much much more brilliant people than I am and they also have similar experiences with Python. So, one last note: every software project goes same tasks as above often much much more officially and carefully, I would suggest managers to see that just do not listen to the ordinary brain-washes. Python is a great choice for easy developing, easy debugging, easy maintaining and most importantly very very time-friendly. Of course there will be tasks .n which Python is suitable, but hey, if it Python is in the list, take it seriously. Thanks, Special thanks to people in this group who had answered my silly Python questions along the way. Sümer Cip -- http://mail.python.org/mailman/listinfo/python-list

1 0

npy/npz file extensions in save/savez
by Ralf Gommers 27 Jun '09

27 Jun '09

Hi, The .npy/.npz extensions are just suggestions and can be replaced with something application-specific according to the NEP (see "Conventions" in http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt). However the code in save/savez does not seem to allow this. Pass in 'filename.ext' and out comes 'filename.ext.npy'. The relevant code in `save` is: if isinstance(file, basestring): if not file.endswith('.npy'): file = file + '.npy' Replacing the second line with: if not os.path.splitext(file)[1]: would fix it I think. Same for `savez`. `load` does the correct thing, it only cares about the magic string and not the extension. Should save/savez allow other extensions? If so, should I open a ticket and attach this as a patch? Cheers, Ralf

1 0

fromfile and ticket #1152
by Charles R Harris 27 Jun '09

27 Jun '09

The question is: what should happen when fewer items are read than requested. The current behaviour is 1) Error message written to stderr (needs to be fixed) 2) If 0 items are read then nomemory error is raised ;) So, should a warning be raised and an array returned with however many items were read? Meaning an empty array if nothing was read. Or should and error be raised in this circumstance? The behaviour is currently undocumented. Chuck

3 2

Re: [Numpy-discussion] switching to float32
by David Goldsmith 27 Jun '09

27 Jun '09

--- On Sat, 6/27/09, Dag Sverre Seljebotn <dagss(a)student.matnat.uio.no> wrote: > Note that it is relatively easy for you to do e.g. > > default_dtype = np.float32 > > def array(*args, **kw): > if 'dtype' not in kw.keys(): > return np.array(*args, **kw, dtype=default_dtype) > else: > return np.array(*args, **kw) Perhaps this could be added to the Cookbook. DG > > and so on in your own codebase, avoiding the problem. > > -- > Dag Sverre > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

1 0

SciPy abstract submission deadline extended
by Gael Varoquaux 27 Jun '09

27 Jun '09

Greetings, The conference committee is extending the deadline for abstract submission for the Scipy conference 2009 one week. On Friday July 3th, at midnight Pacific, we will turn off the abstract submission on the conference site. Up to then, you can modify the already-submitted abstract, or submit new abstracts. Submitting Papers --------------------- The program features tutorials, contributed papers, lightning talks, and bird-of-a-feather sessions. We are soliciting talks and accompanying papers (either formal academic or magazine-style articles) that discuss topics which center around scientific computing using Python. These include applications, teaching, future development directions, and research. A collection of peer-reviewed articles will be published as part of the proceedings. Proposals for talks are submitted as extended abstracts. There are two categories of talks: Paper presentations These talks are 35 minutes in duration (including questions). A one page abstract of no less than 500 words (excluding figures and references) should give an outline of the final paper. Proceeding papers are due two weeks after the conference, and may be in a formal academic style, or in a more relaxed magazine-style format. Rapid presentations These talks are 10 minutes in duration. An abstract of between 300 and 700 words should describe the topic and motivate its relevance to scientific computing. In addition, there will be an open session for lightning talks during which any attendee willing to do so is invited to do a couple-of-minutes-long presentation. If you wish to present a talk at the conference, please create an account on the website (http://conference.scipy.org). You may then submit an abstract by logging in, clicking on your profile and following the "Submit an abstract" link. Submission Guidelines * Submissions should be uploaded via the online form. * Submissions whose main purpose is to promote a commercial product or service will be refused. * All accepted proposals must be presented at the SciPy conference by at least one author. * Authors of an accepted proposal can provide a final paper for publication in the conference proceedings. Final papers are limited to 7 pages, including diagrams, figures, references, and appendices. The papers will be reviewed to help ensure the high-quality of the proceedings. For further information, please visit the conference homepage: http://conference.scipy.org. The SciPy 2009 executive committee ----------------------------------- * Jarrod Millman, UC Berkeley, USA (Conference Chair) * Gaël Varoquaux, INRIA Saclay, France (Program Co-Chair) * Stéfan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair) * Fernando Pérez, UC Berkeley, USA (Tutorial Chair)

1 0

Rec array: numpy.rec vs numpy.array with complex dtype
by Dan Yamins 26 Jun '09

26 Jun '09

Dear Numpy list: We've been using the numpy.rec classes to make record array objects. We've noticed that in more recent versions of numpy, record-array like objects can be made directly with the numpy.ndarray class, by passing a complex data type. However, it looks like the numpy.rec class is still supported. So, we have a couple of questions: 1) Which is the preferred way to make a record array, numpy.rec, or the numpy.ndarray with complex data type? A somewhat detailed explanation of the comparative properties would be great. (We know it's buried somewhere in the document ... sorry for being lazy!) 2) The individual "records" in numpy.rec array have the "numpy.record" type. The individual records in the numpy.array approach have "numpy.void" type. Can you tell us a little about how these differ, and what the advantages of one vs the other is? 3) We've heard talk about "complex data types" in numpy in general. Is there some good place we can read about this more extensively? Also: one thing we use and like about the numpy.rec constructors is that they can take a "names" argument, and the constructor function does some inferring about what the formats you want are, e.g.: img = numpy.rec.fromrecords([(0,1,'a'),(2,0,'b')], names = ['A','B','C']) produces: rec.array([(0, 1, 'a'), (2, 0, 'b')], dtype=[('A', '<i4'), ('B', '<i4'), ('C', '|S1')]) This is very convenient. My immediate guess for the equivalent thing with the numpy.ndarray approach: img = numpy.array([(0,1,'a'),(2,0,'b')], names = ['A','B','C']) does not work. Is there some syntax for doing this? Thanks, Dan

2 3

loading data
by Mag Gam 26 Jun '09

26 Jun '09

Hello. I am very new to NumPy and Python. We are doing some research in our Physics lab and we need to store massive amounts of data (100GB daily). I therefore, am going to use hdf5 and h5py. The problem is I am using np.loadtxt() to create my array and create a dataset according to that. np.loadtxt() is reading a file which is about 50GB. This takes a very long time! I was wondering if there was a much easier and better way of doing this. TIA

4 8

latex equations in docstrings
by Chris Kees 24 Jun '09

24 Jun '09

Hi, Apologies if I sent two copies of this message to the list. If I'm using the math directive in my docstrings following the scipy guidelines (http://projects.scipy.org/numpy/wiki/ CodingStyleGuidelines), what tool do I need to use in order to generate html with properly formated equations? I read that Sphinx is not primarily intended for API docs, but using epydoc I can't seem to get the equations formatted. Are you guys using Sphinx on the numpy/ scipy source? Chris

3 2