From friedrichromstedt at gmail.com Sun Aug 1 05:43:40 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Sun, 1 Aug 2010 11:43:40 +0200 Subject: [Numpy-discussion] str == int puzzlement In-Reply-To: References: Message-ID: 2010/8/1 Matthew Brett : >> Maybe it would be better to raise a ValueError, which is not caught by >> the evaluation mechanism, to prevent such stuff. > > Sorry that this is not yet clear to me, but, is it true then that: > > The only situation where array.__eq__ sensibly falls back to python > __eq__ is for the individual elements of object arrays? Btw, I changed my opinion on that. The rationale was to inform the user about that numpy cannot do that. This is based on that Python cannot do better. But right this assumtion isn't true, since maybe the other operand knows quite well how to compare the pears to the apples, for whatever reason. So better stick to the documentation: "A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments." And since at the moment, the functionality is in fact "NotImplemented", this is the way to go at the moment. Of course, there is room for change, e.g., by breaking it down to the scalar level, if the r.h.s. is not a numpy.ndarray, and returning an array of Falses else. But even this is error-prone, because, the r.h.s. may be a descendant-class instance of numpy.ndarray, knowing better what to do, but being no longer asked. So I come to the conclusion, that in fact all "implementations" of the __eq__ for not-comparable dtypes aren't. They are just based on too many assumptions. They would be a replacement for a real implementation, for a real functionality. To come to your question, Matthew, I think I would tend to give the general recommendation to not mix dtypes which not really compare to each other. It's just error-prone. In fact, when you want to compare by Python's implementation of __eq__, which should in fact be the builtin cmp(), then object arrays are the way to go. But then you cannot use a single non-object ndarray. To summarise, numpy simply does not take part in __eq__ calculations where it does not know what to do. Python does the rest to ask the others about what they think about the operation. But the others should be ndarray-aware. Friedrich From faltet at pytables.org Sun Aug 1 07:33:52 2010 From: faltet at pytables.org (Francesc Alted) Date: Sun, 1 Aug 2010 13:33:52 +0200 Subject: [Numpy-discussion] ANN: Numexpr 1.4 released Message-ID: <201008011333.52074.faltet@pytables.org> Hi, After the Theano talk in last EuroSciPy I suddenly realized that it would not be too difficult to implement a multi-threaded version of Numexpr. Well, as usual I was terribly wrong and it took me a *long* week to do the job :-/ Anyway the thing is done now, so... enjoy! Note for PyTables users: Numexpr does not include changes in the API/ABI, so the upgrade from 1.3.1 is recommended. ======================== Announcing Numexpr 1.4 ======================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. What's new ========== The main improvement in this version is the support for multi-threading in pure C. Threading in C provides the best performance in nowadays multi-core machines. In addition, this avoids the GIL that hampers performance in many Python apps. Just to wet your appetite, look into this page where the implementation is briefly described and where some benchmarks are shown: http://code.google.com/p/numexpr/wiki/MultiThreadVM In case you want to know more in detail what has changed in this version, see: http://code.google.com/p/numexpr/wiki/ReleaseNotes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at Google code in: http://code.google.com/p/numexpr/ And you can get the packages from PyPI as well: http://pypi.python.org/pypi Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy! -- Francesc Alted From sebastian.walter at gmail.com Sun Aug 1 08:05:58 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Sun, 1 Aug 2010 14:05:58 +0200 Subject: [Numpy-discussion] [ANN] ALGOPY 0.21, algorithmic differentiation in Python Message-ID: I'm happy to announce the first official release of ALGOPY in version 0.2.1. Rationale: ~~~~~~~~ The purpose of ALGOPY is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) using univariate Taylor polynomial arithmetic. Particular focus are functions that contain numerical linear algebra functions (e.g. inv, dot, eigh, qr, cholesky,...) as they often appear in statistically motivated functions. Download: ~~~~~~~~~ http://pypi.python.org/pypi/algopy/0.2.1 or bleeding edge versions from from http://github.com/b45ch1/algopy Documentation: ~~~~~~~~~~~~ available at http://packages.python.org/algopy/ OS Support: ~~~~~~~~~~ Linux, Windows (tested with pythonxy), should also work on Mac Software Dependencies: ~~~~~~~~~~~~~~~~~~~~ for the core: numpy, scipy for testing: nose Exampe Session: ~~~~~~~~~~~~~ Consider the contrived example where it is the goal to compute the directional derivative df/dx_1 : >>> import numpy; from numpy import log, exp, sin, cos >>> import algopy; from algopy import UTPM, dot, inv, zeros >>> >>> def f(x): ... A = zeros((2,2),dtype=x) ... A[0,0] = numpy.log(x[0]*x[1]) ... A[0,1] = numpy.log(x[1]) + exp(x[0]) ... A[1,0] = sin(x[1])**2 + cos(x[0])**3.1 ... A[1,1] = x[0]**cos(x[1]) ... return log( dot(x.T, dot( inv(A), x))) ... >>> >>> x = UTPM(zeros((2,1,2),dtype=float)) >>> x.data[0,0] = [1,2] >>> x.data[1,0] = [1,0] >>> y = f(x) >>> >>> print 'normal function evaluation f(x) = ',y.data[0,0] normal function evaluation f(x) = 0.641250189986 >>> print 'directional derivative df/dx1 = ',y.data[1,0] directional derivative df/dx1 = 1.62982340133 From morph at debian.org Sun Aug 1 09:04:45 2010 From: morph at debian.org (Sandro Tosi) Date: Sun, 1 Aug 2010 15:04:45 +0200 Subject: [Numpy-discussion] Numpy 1.4.1 fails to build on (Debian) alpha and powepc In-Reply-To: <4C523FE9.6040406@silveregg.co.jp> References: <4C523FE9.6040406@silveregg.co.jp> Message-ID: Hi David, sorry for the late reply. On Fri, Jul 30, 2010 at 04:58, David wrote: > On 07/30/2010 06:47 AM, Sandro Tosi wrote: > >> For the build logs it's easy: >> >> alpha: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=alpha&ver=1%3A1.4.1-4&stamp=1280296333&file=log&as=raw >> powerpc: https://buildd.debian.org/fetch.cgi?pkg=python-numpy&arch=powerpc&ver=1%3A1.4.1-4&stamp=1280297029&file=log&as=raw >> >> for powerpc "import numpy; numpy.test()" I've already sent you the >> output, want me to re-run them? for alpha, attached you can find the >> log for both 2.5 and 2.6; there are some errors/warnings but nothing >> too dramatic? > > Wow, I am genuily surprised that the alpha test suite has no error (the > 2.5 error has nothing to do with the fixes). :) Anyhow, I plan to enable tests execution at package build time, so that we can spot "strange" behaviors/faults on all the archs supported by Debian. >>> Also, if there is another issue preventing numpy 1.4.x integration on >>> debian and ubuntu, please speak up. Ideally, I would like to remove >> >> I don't think there is anything else (for now :) ) from the numpy >> side: Thanks a lot for the support!! Now on Debian we have to fix some >> packages to avoid breakages when we upgrade numpy in the future (the >> biggest issue was that dtype was extended with new fields at the end, >> but some packages were checking the size of dtype with the one the >> packge was compiled with (1.3.*) and failed). > > Yes, we have improved quite a bit our support here in 1.4.x - we hope > that those issues won't arise in the 1.x series anymore. Note also that > if those dtype errors appear with pyrex/cython-generated code, using a > more recent cython will prevent the error from happening (warnings > raised instead). Ah, thanks for letting me know! I'll keep in mind. >> As usual, Ubuntu will >> just sit and wait for us to do the work and then just sync it (sigh). >> >>> the need for downstream patches (none of them was necessary IIRC), >> >> Here is the list of the patches we currently have in the Debian >> package (you can look at them at [2]): >> >> 02_build_dotblas.patch >> - Patch to build _dotblas.c when ATLAS is not installed. >> -- dunno exactly what it does, it seems to infer _dotblas is compiled >> is ATLAS is missing > > This is is caused by not having atlas as a build dependency I guess. > Strictly speaking, dotblas only requires cblas, but we don't have the > check in place to do so. Since numscons already does this, and it has > worked pretty well, maybe I will take time to add this as well in > numpy.distutils. But this has relatively little consequence I think. yep, I agree, it would be nice tho to have it directly upstream. >> >> 03_force_f2py_version.patch >> - force generation f2py postfixed with interpreter version >> -- Debian specific: we ship f2py2.5 and f2py2.6 and we make f2py a >> symlink towards f2py2.6 > > ok. > > >> 05_fix_endianness_detection.patch >> - Fix endianness detection: endian.h should be present on all Debian >> machines. This patch forces the use of endian.h, this reventing >> several reverse dependencies os Numpy from failing to build. >> -- Debian specific: we want to enforce the usage of endian.h file >> available on all of our architectures > > This one has been fixed in the 1.4.x branch (and trunk of course) > >> >> 07_bts585309_string_exceptions.diff >> - Remove string exceptions >> -- patch from trunk, we can remove it once a new release is out > > This one as well I'll live those patches there, until a new release will be out, or do you suggest to sync the package with the 1.4.x branch? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From ralf.gommers at googlemail.com Sun Aug 1 12:38:33 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 2 Aug 2010 00:38:33 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 Message-ID: I am pleased to announce the availability of the first beta of NumPy 1.5.0. This will be the first NumPy release to include support for Python 3, as well as for Python 2.7. Please try this beta and report any problems on the NumPy mailing list. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ Please note that binaries for Python 3.1 are not yet up, they will follow as soon as a minor issue with building them is resolved. Building from source with Python 3.1 should work without problems. Enjoy, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Sun Aug 1 14:20:24 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 1 Aug 2010 20:20:24 +0200 Subject: [Numpy-discussion] ANN: Numexpr 1.4 released In-Reply-To: <201008011333.52074.faltet@pytables.org> References: <201008011333.52074.faltet@pytables.org> Message-ID: Congratulations Francesc ! On Sun, Aug 1, 2010 at 1:33 PM, Francesc Alted wrote: > Hi, > > After the Theano talk in last EuroSciPy I suddenly realized that it would not > be too difficult to implement a multi-threaded version of Numexpr. ?Well, as > usual I was terribly wrong and it took me a *long* week to do the job :-/ this is really exciting !! Would you mind summarizing very briefly which dtypes you support in Numexpr ? (float32?) And 2ndly, is there (regarding this) a difference between the Numexpr from pyTables and the "normal" Numexpr (which I think lives at google code these days) ... ? Thanks, Sebastian Haase From jsalvati at u.washington.edu Sun Aug 1 14:34:05 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Sun, 1 Aug 2010 11:34:05 -0700 Subject: [Numpy-discussion] Mathematical language for reasoning about numpy like arrays? Message-ID: Is anyone aware of a good mathematical language to describe and reason about how numpy arrays work (broadcastable ndarrays)? I am particularly interested in reasoning about linear maps from one ndarray to another. Clearly ndarrays can be thought of as regular vectors and you can reason about them using linear algebra, but their ndimensionality and broadcasting also gives them more structure which gives them some interesting properties which I would like to reason about. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Sun Aug 1 14:57:59 2010 From: faltet at pytables.org (Francesc Alted) Date: Sun, 1 Aug 2010 20:57:59 +0200 Subject: [Numpy-discussion] ANN: Numexpr 1.4 released In-Reply-To: References: <201008011333.52074.faltet@pytables.org> Message-ID: Hey Sebastian, 2010/8/1 Sebastian Haase > Would you mind summarizing very briefly which dtypes you support in Numexpr > ? > (float32?) > Yup. See: http://code.google.com/p/numexpr/wiki/Overview#Datatypes_supported_internally for the complete list. > And 2ndly, is there (regarding this) a difference between the Numexpr > from pyTables and the "normal" Numexpr (which I think lives at google > code these days) ... ? > Since 2.2, PyTables does not includes Numexpr anymore and depends on Numexpr (>= 1.3.1) explicitly. The reason is that, after "normal" Numexpr got all the enhancements that PyTables needed (mainly support for booleans and strided and unaligned data), it does not make sense to have different Numexpr's anymore. Cheers, -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Sun Aug 1 15:05:10 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 01 Aug 2010 12:05:10 -0700 Subject: [Numpy-discussion] ANN: Numexpr 1.4 released In-Reply-To: <201008011333.52074.faltet@pytables.org> References: <201008011333.52074.faltet@pytables.org> Message-ID: <4C55C566.9090005@uci.edu> Solid release as usual. Works well with the MKL. Btw, numexpr-1.4.tar.gz is missing the win32/pthread.h file. Christoph On 8/1/2010 4:33 AM, Francesc Alted wrote: > Hi, > > After the Theano talk in last EuroSciPy I suddenly realized that it would not > be too difficult to implement a multi-threaded version of Numexpr. Well, as > usual I was terribly wrong and it took me a *long* week to do the job :-/ > > Anyway the thing is done now, so... enjoy! > > Note for PyTables users: Numexpr does not include changes in the API/ABI, so > the upgrade from 1.3.1 is recommended. > > ======================== > Announcing Numexpr 1.4 > ======================== > > Numexpr is a fast numerical expression evaluator for NumPy. With it, > expressions that operate on arrays (like "3*a+4*b") are accelerated > and use less memory than doing the same calculation in Python. > > What's new > ========== > > The main improvement in this version is the support for > multi-threading in pure C. Threading in C provides the best > performance in nowadays multi-core machines. In addition, this avoids > the GIL that hampers performance in many Python apps. > > Just to wet your appetite, look into this page where the > implementation is briefly described and where some benchmarks are > shown: > > http://code.google.com/p/numexpr/wiki/MultiThreadVM > > In case you want to know more in detail what has changed in this > version, see: > > http://code.google.com/p/numexpr/wiki/ReleaseNotes > > or have a look at RELEASE_NOTES.txt in the tarball. > > Where I can find Numexpr? > ========================= > > The project is hosted at Google code in: > > http://code.google.com/p/numexpr/ > > And you can get the packages from PyPI as well: > > http://pypi.python.org/pypi > > Share your experience > ===================== > > Let us know of any bugs, suggestions, gripes, kudos, etc. you may > have. > > > Enjoy! > > From faltet at pytables.org Sun Aug 1 15:19:34 2010 From: faltet at pytables.org (Francesc Alted) Date: Sun, 1 Aug 2010 21:19:34 +0200 Subject: [Numpy-discussion] ANN: Numexpr 1.4 released In-Reply-To: <4C55C566.9090005@uci.edu> References: <201008011333.52074.faltet@pytables.org> <4C55C566.9090005@uci.edu> Message-ID: 2010/8/1 Christoph Gohlke > Solid release as usual. Works well with the MKL. > > Btw, numexpr-1.4.tar.gz is missing the win32/pthread.h file. > Mmh, not so solid ;-) Fixed. Thanks for reporting! -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Sun Aug 1 18:16:47 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Sun, 1 Aug 2010 15:16:47 -0700 Subject: [Numpy-discussion] [ANN] ALGOPY 0.21, algorithmic differentiation in Python In-Reply-To: References: Message-ID: Holy cow! I was looking for this exact package for extending pymc! Now I've found two packages that do basically exactly what I want (Theano and ALGOPY). Does ALYGOPY handle derivatives of operations on higher order ndimensional arrays well (efficiently and including broadcasting and such)? John On Sun, Aug 1, 2010 at 5:05 AM, Sebastian Walter wrote: > I'm happy to announce the first official release of ALGOPY in version > 0.2.1. > > Rationale: > ~~~~~~~~ > The purpose of ALGOPY is the evaluation of higher-order derivatives in > the forward and reverse mode of Algorithmic Differentiation (AD) using > univariate Taylor polynomial arithmetic. Particular focus are > functions that contain numerical linear algebra functions (e.g. inv, > dot, eigh, qr, cholesky,...) as they often appear in statistically > motivated functions. > > Download: > ~~~~~~~~~ > http://pypi.python.org/pypi/algopy/0.2.1 > or bleeding edge versions from from http://github.com/b45ch1/algopy > > Documentation: > ~~~~~~~~~~~~ > available at http://packages.python.org/algopy/ > > OS Support: > ~~~~~~~~~~ > Linux, Windows (tested with pythonxy), should also work on Mac > > Software Dependencies: > ~~~~~~~~~~~~~~~~~~~~ > for the core: numpy, scipy > for testing: nose > > Exampe Session: > ~~~~~~~~~~~~~ > Consider the contrived example where it is the goal to compute the > directional derivative df/dx_1 : > > >>> import numpy; from numpy import log, exp, sin, cos > >>> import algopy; from algopy import UTPM, dot, inv, zeros > >>> > >>> def f(x): > ... A = zeros((2,2),dtype=x) > ... A[0,0] = numpy.log(x[0]*x[1]) > ... A[0,1] = numpy.log(x[1]) + exp(x[0]) > ... A[1,0] = sin(x[1])**2 + cos(x[0])**3.1 > ... A[1,1] = x[0]**cos(x[1]) > ... return log( dot(x.T, dot( inv(A), x))) > ... > >>> > >>> x = UTPM(zeros((2,1,2),dtype=float)) > >>> x.data[0,0] = [1,2] > >>> x.data[1,0] = [1,0] > >>> y = f(x) > >>> > >>> print 'normal function evaluation f(x) = ',y.data[0,0] > normal function evaluation f(x) = 0.641250189986 > >>> print 'directional derivative df/dx1 = ',y.data[1,0] > directional derivative df/dx1 = 1.62982340133 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Sun Aug 1 19:10:27 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sun, 01 Aug 2010 19:10:27 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: <4C55FEE3.40303@gmail.com> On 8/1/2010 12:38 PM, Ralf Gommers wrote: > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ I'm not seeing them. Alan From olivier.grisel at ensta.org Sun Aug 1 19:11:23 2010 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Mon, 2 Aug 2010 01:11:23 +0200 Subject: [Numpy-discussion] [ANN] ALGOPY 0.21, algorithmic differentiation in Python In-Reply-To: References: Message-ID: 2010/8/2 John Salvatier : > Holy cow! I was looking for this exact package for extending pymc! Now I've > found two packages that do basically exactly what I want (Theano and > ALGOPY). Beware that theano does only symbolic differentiation which is very different from AD. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From ralf.gommers at googlemail.com Sun Aug 1 22:10:59 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 2 Aug 2010 10:10:59 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: <4C55FEE3.40303@gmail.com> References: <4C55FEE3.40303@gmail.com> Message-ID: On Mon, Aug 2, 2010 at 7:10 AM, Alan G Isaac wrote: > On 8/1/2010 12:38 PM, Ralf Gommers wrote: > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ < > https://sourceforge.net/projects/numpy/files/> > > I'm not seeing them. > > I see them now. Normally SF updates in <15min, this time it took half a day for some reason. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Sun Aug 1 22:18:57 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 1 Aug 2010 21:18:57 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: <4C55FEE3.40303@gmail.com> Message-ID: On Sun, Aug 1, 2010 at 9:10 PM, Ralf Gommers wrote: > > > On Mon, Aug 2, 2010 at 7:10 AM, Alan G Isaac wrote: >> >> On 8/1/2010 12:38 PM, Ralf Gommers wrote: >> > Binaries, sources and release notes can be found at >> > https://sourceforge.net/projects/numpy/files/ >> > >> >> I'm not seeing them. >> > I see them now. Normally SF updates in <15min, this time it took half a day > for some reason. > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Hi, I get a lot of umath_complex related problems with Python 2.6.3 32-bit on 64-bit Windows 7. Bruce Python 2.6.3 (r263rc1:75186, Oct 2 2009, 20:40:30) [MSC v.1500 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 2.6.3 >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.5.0b1 NumPy is installed in E:\Python26\lib\site-packages\numpy Python version 2.6.3 (r263rc1:75186, Oct 2 2009, 20:40:30) [MSC v.1500 32 bit (Intel)] nose version 0.11.1 ..............................................................................................................E........................................................................................................................................................................................................................................................................................................................................................................K................................................................................................................................................................................................................................K..........................................................K..K..............................K........F....FFFF.....K.F.........FF..FFF.....F................................................................................S..................................................................................................................................................................................................................................................................................................................................................................................................................................................K.........K..........................................................................................................................................................................................................................................................................................................................................................................................................................................................S........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ====================================================================== ERROR: test_filename (test_memmap.TestMemmap) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\numpy\core\tests\test_memmap.py", line 60, in test_filename os.unlink(tmpname) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'e:\\users\\me\\appdata\\local\\temp\\mmapklcldk' ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(, inf, 0, inf, 0) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, in check_complex_value assert_equal(f(z1), z2) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ inf nanj]) y: array((inf+0j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True], False mismatch)\n x: array([ inf nanj])\n y: array((inf+0j))') ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 60, in _check_ninf_inf raise AssertionError(msgform %(z.real, z.imag)) AssertionError: cexp(-inf, inf) is (-1.#IND00, -1.#IND00), expected (+-0, +-0) >> raise AssertionError('cexp(-inf, inf) is (%f, %f), expected (+-0, +-0)' %((nan+nan*j).real, (nan+nan*j).imag)) ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 73, in _check_inf_inf raise AssertionError(msgform % (z.real, z.imag)) AssertionError: cexp(inf, inf) is (-1.#IND00, -1.#IND00), expected (+-inf, nan) >> raise AssertionError('cexp(inf, inf) is (%f, %f), expected (+-inf, nan)' % ((nan+nan*j).real, (nan+nan*j).imag)) ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 86, in _check_ninf_nan raise AssertionError(msgform % (z.real, z.imag)) AssertionError: cexp(-inf, nan) is (1.#QNAN0, 1.#QNAN0), expected (+-0, +-0) >> raise AssertionError('cexp(-inf, nan) is (%f, %f), expected (+-0, +-0)' % ((nan+nan*j).real, (nan+nan*j).imag)) ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 99, in _check_inf_nan raise AssertionError(msgform % (z.real, z.imag)) AssertionError: cexp(-inf, nan) is (1.#QNAN0, 1.#QNAN0), expected (+-inf, nan) >> raise AssertionError('cexp(-inf, nan) is (%f, %f), expected (+-inf, nan)' % ((nan+nan*j).real, (nan+nan*j).imag)) ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 144, in test_special_values self.assertRaises(FloatingPointError, np.log, x) AssertionError: FloatingPointError not raised if hasattr(excClass,'__name__'): 'FloatingPointError' = excClass.'unittest' else: 'FloatingPointError' = str(excClass) >> raise self.failureException, "%s not raised" % 'FloatingPointError' ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, in check_complex_value assert_equal(f(z1), z2) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ inf nanj]) y: array((inf+inf*j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True], False mismatch)\n x: array([ inf nanj])\n y: array((inf+inf*j))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, in check_complex_value assert_equal(f(z1), z2) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ nan infj]) y: array((inf+inf*j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True], False mismatch)\n x: array([ nan infj])\n y: array((inf+inf*j))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, in check_complex_value assert_equal(f(z1), z2) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ inf nanj]) y: array((inf+inf*j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True], False mismatch)\n x: array([ inf nanj])\n y: array((inf+inf*j))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -inf, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, in check_complex_value assert_equal(f(z1), z2) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ nan infj]) y: array((inf+inf*j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True], False mismatch)\n x: array([ nan infj])\n y: array((inf+inf*j))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, in check_complex_value assert_equal(f(z1), z2) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "E:\Python26\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ nan nanj]) y: array((inf+inf*j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True], False mismatch)\n x: array([ nan nanj])\n y: array((inf+inf*j))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line 183, in runTest self.test(*self.arg) File "E:\Python26\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 340, in _check_ninf_nan raise AssertionError(msgform % (z.real, z.imag)) AssertionError: csqrt(-inf, nan) is (1.#QNAN0, 1.#QNAN0), expected (nan, +-inf) >> raise AssertionError('csqrt(-inf, nan) is (%f, %f), expected (nan, +-inf)' % ((nan+nan*j).real, (nan+nan*j).imag)) ---------------------------------------------------------------------- Ran 2456 tests in 30.725s FAILED (KNOWNFAIL=8, SKIP=2, errors=1, failures=12) >>> From alan.isaac at gmail.com Sun Aug 1 22:30:31 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sun, 01 Aug 2010 22:30:31 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: <4C55FEE3.40303@gmail.com> Message-ID: <4C562DC7.3070109@gmail.com> Tests produce a few failures and a couple warnings. Details below. Alan Isaac Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test() Running unit tests for numpy NumPy version 1.5.0b1 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] nose version 0.11.0 .......................................................................................... ....................E..................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .....................K.................................................................... .......................................................................................... ..................................................................K....................... ...................................K..K.............Warning: invalid value encountered in absolute .Warning: invalid value encountered in absolute ................K........F....FFFF.....K.F.........FF..FFF.....F.......................... ......................................................S................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................K.........K........................................ .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ..........................................S............................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ................................................................ ====================================================================== ERROR: test_filename (test_memmap.TestMemmap) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\numpy\core\tests\test_memmap.py", line 60, in test_f ilename os.unlink(tmpname) WindowsError: [Error 32] The process cannot access the file because it is being used by an other process: 'c:\\users\\alanis~1\\appdata\\local\\temp\\mmapoo5c2b' ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(, inf, 0, inf, 0) test_umath_complex.TestCexp.test_special_values(, inf, 0, inf, 0) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, i n check_complex_value assert_equal(f(z1), z2) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_e qual verbose=verbose, header='Arrays are not equal') File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_c ompare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ inf nanj]) y: array((inf+0j)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True] , False mismatch)\n x: array([ inf nanj])\n y: array((inf+0j))') ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 60, in _check_ninf_inf raise AssertionError(msgform %(z.real, z.imag)) AssertionError: cexp(-inf, inf) is (nan, nan), expected (+-0, +-0) >> raise AssertionError('cexp(-inf, inf) is (%f, %f), expected (+-0, +-0)' %((nan+nan*j). real, (nan+nan*j).imag)) ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 73, in _check_inf_inf raise AssertionError(msgform % (z.real, z.imag)) AssertionError: cexp(inf, inf) is (nan, nan), expected (+-inf, nan) >> raise AssertionError('cexp(inf, inf) is (%f, %f), expected (+-inf, nan)' % ((nan+nan*j ).real, (nan+nan*j).imag)) ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 86, in _check_ninf_nan raise AssertionError(msgform % (z.real, z.imag)) AssertionError: cexp(-inf, nan) is (nan, nan), expected (+-0, +-0) >> raise AssertionError('cexp(-inf, nan) is (%f, %f), expected (+-0, +-0)' % ((nan+nan*j) .real, (nan+nan*j).imag)) ====================================================================== FAIL: test_umath_complex.TestCexp.test_special_values(None,) test_umath_complex.TestCexp.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 99, in _check_inf_nan raise AssertionError(msgform % (z.real, z.imag)) AssertionError: cexp(-inf, nan) is (nan, nan), expected (+-inf, nan) >> raise AssertionError('cexp(-inf, nan) is (%f, %f), expected (+-inf, nan)' % ((nan+nan* j).real, (nan+nan*j).imag)) ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 144, i n test_special_values self.assertRaises(FloatingPointError, np.log, x) AssertionError: FloatingPointError not raised ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, inf, inf, inf) test_umath_complex.TestCsqrt.test_special_values(, 1, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, i n check_complex_value assert_equal(f(z1), z2) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_e qual verbose=verbose, header='Arrays are not equal') File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_c ompare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ inf nanj]) y: array((inf+infj)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True] , False mismatch)\n x: array([ inf nanj])\n y: array((inf+infj))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, inf, inf, inf) test_umath_complex.TestCsqrt.test_special_values(, -1, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, i n check_complex_value assert_equal(f(z1), z2) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_e qual verbose=verbose, header='Arrays are not equal') File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_c ompare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ nan infj]) y: array((inf+infj)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True] , False mismatch)\n x: array([ nan infj])\n y: array((inf+infj))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, inf, inf, inf) test_umath_complex.TestCsqrt.test_special_values(, inf, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, i n check_complex_value assert_equal(f(z1), z2) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_e qual verbose=verbose, header='Arrays are not equal') File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_c ompare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ inf nanj]) y: array((inf+infj)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True] , False mismatch)\n x: array([ inf nanj])\n y: array((inf+infj))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -inf, inf, inf, inf ) test_umath_complex.TestCsqrt.test_special_values(, -inf, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, i n check_complex_value assert_equal(f(z1), z2) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_e qual verbose=verbose, header='Arrays are not equal') File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_c ompare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ nan infj]) y: array((inf+infj)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True] , False mismatch)\n x: array([ nan infj])\n y: array((inf+infj))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, inf, inf, inf) test_umath_complex.TestCsqrt.test_special_values(, nan, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 539, i n check_complex_value assert_equal(f(z1), z2) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, in assert_array_e qual verbose=verbose, header='Arrays are not equal') File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, in assert_array_c ompare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ nan nanj]) y: array((inf+infj)) >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location mismatch [ True] , False mismatch)\n x: array([ nan nanj])\n y: array((inf+infj))') ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(None,) test_umath_complex.TestCsqrt.test_special_values(None,) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 340, i n _check_ninf_nan raise AssertionError(msgform % (z.real, z.imag)) AssertionError: csqrt(-inf, nan) is (nan, nan), expected (nan, +-inf) >> raise AssertionError('csqrt(-inf, nan) is (%f, %f), expected (nan, +-inf)' % ((nan+nan *j).real, (nan+nan*j).imag)) ---------------------------------------------------------------------- Ran 2456 tests in 19.070s FAILED (KNOWNFAIL=8, SKIP=2, errors=1, failures=12) >>> From sebastian.walter at gmail.com Mon Aug 2 02:20:51 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Mon, 2 Aug 2010 08:20:51 +0200 Subject: [Numpy-discussion] [ANN] ALGOPY 0.21, algorithmic differentiation in Python In-Reply-To: References: Message-ID: On Mon, Aug 2, 2010 at 12:16 AM, John Salvatier wrote: > Holy cow! I was looking for this exact package for extending pymc! Now I've > found two packages that do basically exactly what I want (Theano and > ALGOPY). > > Does ALYGOPY handle derivatives of operations on higher order ndimensional > arrays well (efficiently and including broadcasting and such)? Yes, no problem. operations on multidimensional arrays: -------------------------------------------------------- In [6]: import numpy In [7]: import algopy In [8]: x = algopy.UTPM(numpy.ones((2,1,2,3,4,5,6))) In [9]: x.shape Out[9]: (2, 3, 4, 5, 6) In [10]: x.data.shape Out[10]: (2, 1, 2, 3, 4, 5, 6) In [11]: y = numpy.sin(x) In [12]: y.shape Out[12]: (2, 3, 4, 5, 6) In [13]: y.data.shape Out[13]: (2, 1, 2, 3, 4, 5, 6) broadcasting ------------------- In [14]: z = algopy.UTPM(numpy.ones((2,1,4,1,6))) In [15]: y = x*z In [16]: z.shape Out[16]: (4, 1, 6) In [17]: y.shape Out[17]: (2, 3, 4, 5, 6) Sebastian > > John > > On Sun, Aug 1, 2010 at 5:05 AM, Sebastian Walter > wrote: >> >> I'm happy to announce the first official release of ALGOPY in version >> 0.2.1. >> >> Rationale: >> ~~~~~~~~ >> The purpose of ALGOPY is the evaluation of higher-order derivatives in >> the forward and reverse mode of Algorithmic Differentiation (AD) using >> univariate Taylor polynomial arithmetic. Particular focus are >> functions that contain numerical linear algebra functions (e.g. inv, >> dot, eigh, qr, cholesky,...) as they often appear in statistically >> motivated functions. >> >> Download: >> ~~~~~~~~~ >> http://pypi.python.org/pypi/algopy/0.2.1 >> or bleeding edge versions from from http://github.com/b45ch1/algopy >> >> Documentation: >> ~~~~~~~~~~~~ >> available at http://packages.python.org/algopy/ >> >> OS Support: >> ~~~~~~~~~~ >> Linux, Windows (tested with pythonxy), should also work on Mac >> >> Software Dependencies: >> ~~~~~~~~~~~~~~~~~~~~ >> for the core: numpy, scipy >> for testing: ?nose >> >> Exampe Session: >> ~~~~~~~~~~~~~ >> Consider the contrived example where it is the goal to compute the >> directional derivative df/dx_1 : >> >> >>> import numpy; from numpy import log, exp, sin, cos >> >>> import algopy; from algopy import UTPM, dot, inv, zeros >> >>> >> >>> def f(x): >> ... ? ? A = zeros((2,2),dtype=x) >> ... ? ? A[0,0] = numpy.log(x[0]*x[1]) >> ... ? ? A[0,1] = numpy.log(x[1]) + exp(x[0]) >> ... ? ? A[1,0] = sin(x[1])**2 + cos(x[0])**3.1 >> ... ? ? A[1,1] = x[0]**cos(x[1]) >> ... ? ? return log( dot(x.T, ?dot( inv(A), x))) >> ... >> >>> >> >>> x = UTPM(zeros((2,1,2),dtype=float)) >> >>> x.data[0,0] = [1,2] >> >>> x.data[1,0] = [1,0] >> >>> y = f(x) >> >>> >> >>> print 'normal function evaluation f(x) = ',y.data[0,0] >> normal function evaluation f(x) = ?0.641250189986 >> >>> print 'directional derivative df/dx1 = ',y.data[1,0] >> directional derivative df/dx1 = ?1.62982340133 >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ralf.gommers at googlemail.com Mon Aug 2 11:48:52 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 2 Aug 2010 23:48:52 +0800 Subject: [Numpy-discussion] distutils issue - python 3.1 on windows Message-ID: Hi, I'm trying to get building to work with Python 3.1 under Wine on OS X. The first thing you run into is a python distutils problem, which is fixed by replacing line 379 of cygwinccompiler.py with result = RE_VERSION.search(str(out_string)) The next thing I run into is a numpy.distutils issue (complete build output below email): File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\mingw32ccompiler.py", line 177, in link func(*args[:func.__func__.__code__.co_argcount]) AttributeError: 'function' object has no attribute '__func__' What I think that line is for is to test what kind of arguments can be passed to 'func', but I'm really not sure. A comment would have been helpful, that must be the most obscure line of Python code I've ever seen:) Does anyone have any idea what causes this error? Thanks, Ralf <... 2to3 output ...> F2PY Version 1 Running from numpy source directory.numpy\core\code_generators\generate_umath.py:117: DeprecationWarning: string.maketrans is deprecated, use bytes.maketrans instead bytes(string.ascii_uppercase, "ascii")) blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in C:\Python31\lib libraries mkl,vml,guide not found in C:\ libraries mkl,vml,guide not found in C:\Python31\libs NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in C:\Python31\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in C:\Python31\libs NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in C:\Python31\lib libraries f77blas,cblas,atlas not found in C:\ libraries f77blas,cblas,atlas not found in C:\Python31\libs NOT AVAILABLE Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\system_info.py:1399: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: FOUND: libraries = ['blas'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in C:\Python31\lib libraries mkl,vml,guide not found in C:\ libraries mkl,vml,guide not found in C:\Python31\libs NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in C:\Python31\lib libraries lapack_atlas not found in C:\Python31\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in C:\Python31\libs libraries lapack_atlas not found in C:\Python31\libs NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in C:\Python31\lib libraries lapack_atlas not found in C:\Python31\lib libraries f77blas,cblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries f77blas,cblas,atlas not found in C:\Python31\libs libraries lapack_atlas not found in C:\Python31\libs NOT AVAILABLE Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\system_info.py:1306: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['C:\\local\\lib\\yop\\nosse'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 could not resolve pattern in '': '*.txt' non-existing path in '': 'COMPATIBILITY' non-existing path in '': 'site.cfg.example' running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build\src.win32-3.1 creating build\src.win32-3.1\numpy creating build\src.win32-3.1\numpy\distutils building library "npymath" sources gcc -dumpversion ld -v dllwrap --version customize GnuFCompiler Found executable C:\MinGW\bin\g77.exe gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found Found executable C:\MinGW\bin\g77.exe customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes compile options: '-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Python31\include -IC:\Python31\PC -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Python31\include -IC:\Python31\PC -c _configtest.c -o _configtest.o Found executable C:\MinGW\bin\gcc.exe Traceback (most recent call last): File "setup.py", line 210, in setup_package() File "setup.py", line 203, in setup_package configuration=configuration ) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "C:\Python31\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Python31\lib\distutils\dist.py", line 919, in run_commands self.run_command(cmd) File "C:\Python31\lib\distutils\dist.py", line 938, in run_command cmd_obj.run() File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "C:\Python31\lib\distutils\command\build.py", line 128, in run self.run_command(cmd_name) File "C:\Python31\lib\distutils\cmd.py", line 315, in run_command self.distribution.run_command(command) File "C:\Python31\lib\distutils\dist.py", line 938, in run_command cmd_obj.run() File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 678, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "C:\Python31\lib\distutils\command\config.py", line 248, in try_link libraries, library_dirs, lang) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\config.py", line 149, in _link libraries, library_dirs, lang)) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\command\config.py", line 89, in _wrap_method ret = mth(*((self,)+args)) File "C:\Python31\lib\distutils\command\config.py", line 145, in _link target_lang=lang) File "C:\Python31\lib\distutils\ccompiler.py", line 808, in link_executable debug, extra_preargs, extra_postargs, None, target_lang) File "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils\mingw32ccompiler.py", line 177, in link func(*args[:func.__func__.__code__.co_argcount]) AttributeError: 'function' object has no attribute '__func__' Captured Task Output: --------------------- ---> pavement.bdist_superpack Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", line 164, in _run_task return do_task() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Paver-1.0.1-py2.6.egg/paver/tasks.py", line 161, in do_task return func(**kw) File "pavement.py", line 213, in bdist_superpack bdist_wininst_arch(pyver, 'nosse') File "pavement.py", line 196, in bdist_wininst_arch _bdist_wininst(pyver, SITECFG[arch]) File "pavement.py", line 266, in _bdist_wininst subprocess.check_call(cmd, env=cfg_env) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 488, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['wine', '/Users/rgommers/.wine/drive_c/Python31/python.exe', 'setup.py', 'build', '-c', 'mingw32', 'bdist_wininst']' returned non-zero exit status 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Mon Aug 2 13:24:33 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 2 Aug 2010 10:24:33 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: When I try to build in 3.1.1 (on Ubuntu 9.10) straight from numpy-1.5.0b1.tar.gz, I get an "ImportError: No module named py3tool" - is this an external dependency, or is it supposed to be included in the source distribution? On Sun, Aug 1, 2010 at 9:38 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the first beta of NumPy 1.5.0. > This will be the first NumPy release to include support for Python 3, as > well as for Python 2.7. Please try this beta and report any problems on the > NumPy mailing list. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > Please note that binaries for Python 3.1 are not yet up, they will follow as > soon as a minor issue with building them is resolved. Building from source > with Python 3.1 should work without problems. > > Enjoy, > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Erik Tollerud From pav at iki.fi Mon Aug 2 13:42:31 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 2 Aug 2010 17:42:31 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 References: Message-ID: Mon, 02 Aug 2010 10:24:33 -0700, Erik Tollerud wrote: > When I try to build in 3.1.1 (on Ubuntu 9.10) straight from > numpy-1.5.0b1.tar.gz, I get an "ImportError: No module named py3tool" - > is this an external dependency, or is it supposed to be included in the > source distribution? It's supposed to be included in the source distribution -- tools/py3tool.py seems to be missing from the tarball. -- Pauli Virtanen From pav at iki.fi Mon Aug 2 13:48:37 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 2 Aug 2010 17:48:37 +0000 (UTC) Subject: [Numpy-discussion] distutils issue - python 3.1 on windows References: Message-ID: Mon, 02 Aug 2010 23:48:52 +0800, Ralf Gommers wrote: > I'm trying to get building to work with Python 3.1 under Wine on OS X. > The first thing you run into is a python distutils problem, which is > fixed by replacing line 379 of cygwinccompiler.py with > result = RE_VERSION.search(str(out_string)) That's going to bust when out_string contains stuff that does not fit in ASCII. I don't remember how this was addressed in the rest of distutils. > The next thing I run into is a numpy.distutils issue (complete build > output below email): > > File > "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils \mingw32ccompiler.py", > line 177, in link > func(*args[:func.__func__.__code__.co_argcount]) > AttributeError: 'function' object has no attribute '__func__' > > What I think that line is for is to test what kind of arguments can be > passed to 'func', but I'm really not sure. A comment would have been > helpful, that must be the most obscure line of Python code I've ever > seen:) Does anyone have any idea what causes this error? This is change in the code object internals in Python 3, and 2to3 doesn't seem able make the conversions automatically. I don't remember exactly how it goes, but there are other instances of this in Numpy -- need to do one thing on Python 2 and another on Python 3. Anyway, here it seems like the `inspect` module should be used -- no need to go mucking with in the code objects to just find the number of arguments for a function. Nobody has tested this part of the distutils code on Python 3, and indeed it does not have any tests, so it's not a surprise that stuff like this is left over :) -- Pauli Virtanen From robert.kern at gmail.com Mon Aug 2 13:52:12 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 2 Aug 2010 12:52:12 -0500 Subject: [Numpy-discussion] distutils issue - python 3.1 on windows In-Reply-To: References: Message-ID: On Mon, Aug 2, 2010 at 12:48, Pauli Virtanen wrote: > Mon, 02 Aug 2010 23:48:52 +0800, Ralf Gommers wrote: >> I'm trying to get building to work with Python 3.1 under Wine on OS X. >> The first thing you run into is a python distutils problem, which is >> fixed by replacing line 379 of ?cygwinccompiler.py with >> ? ? result = RE_VERSION.search(str(out_string)) > > That's going to bust when out_string contains stuff that does not fit in > ASCII. > > I don't remember how this was addressed in the rest of distutils. > >> The next thing I run into is a numpy.distutils issue (complete build >> output below email): >> >> ? File >> "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils > \mingw32ccompiler.py", >> line 177, in link >> ? ? func(*args[:func.__func__.__code__.co_argcount]) >> AttributeError: 'function' object has no attribute '__func__' >> >> What I think that line is for is to test what kind of arguments can be >> passed to 'func', but I'm really not sure. A comment would have been >> helpful, that must be the most obscure line of Python code I've ever >> seen:) Does anyone have any idea what causes this error? > > This is change in the code object internals in Python 3, and 2to3 doesn't > seem able make the conversions automatically. I don't remember exactly > how it goes, but there are other instances of this in Numpy -- need to do > one thing on Python 2 and another on Python 3. > > Anyway, here it seems like the `inspect` module should be used -- no need > to go mucking with in the code objects to just find the number of > arguments for a function. > > Nobody has tested this part of the distutils code on Python 3, and indeed > it does not have any tests, so it's not a surprise that stuff like this > is left over :) I believe we avoided the inspect module because it is quite expensive to import. It may not matter inside numpy.distutils, but be wary of "fixing" things to use inspect elsewhere. It would be worth extracting the commonly-used pieces of inspect (and hacks like this) into an internal utility module that is fast to import. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Mon Aug 2 13:57:41 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 2 Aug 2010 17:57:41 +0000 (UTC) Subject: [Numpy-discussion] distutils issue - python 3.1 on windows References: Message-ID: Mon, 02 Aug 2010 12:52:12 -0500, Robert Kern wrote: [clip] > I believe we avoided the inspect module because it is quite expensive to > import. It may not matter inside numpy.distutils, but be wary of > "fixing" things to use inspect elsewhere. It would be worth extracting > the commonly-used pieces of inspect (and hacks like this) into an > internal utility module that is fast to import. We actually have `numpy.compat._inspect` and from numpy.compat import getargspec that could be used here. -- Pauli Virtanen From matt.gregory at oregonstate.edu Mon Aug 2 14:14:29 2010 From: matt.gregory at oregonstate.edu (Gregory, Matthew) Date: Mon, 2 Aug 2010 11:14:29 -0700 Subject: [Numpy-discussion] accumulated sum-of-squared-differences Message-ID: <1D673F86DDA00841A1216F04D1CE70D641A8C9435B@EXCH2.nws.oregonstate.edu> Hi all, I'm trying to calculate accumulated sum-of-squared-differences for an array in the following manner: import numpy as np a = np.array([1, 2, 3, 49., 50, 51, 98, 99, 100], dtype=np.float32) # Calculate accumulated means over all elements means = np.add.accumulate(a) / (np.arange(a.size) + 1) # Create a matrix of squared differences (elements minus means) diff_sqr = np.asarray(np.matrix(a) - np.matrix(means).T) ** 2 # Sum the lower triangular elements ssd = np.tril(diff_sqr).sum(axis=1) Is there an easier or more terse way to do this? Ideally, I'd like to accumulate over all slices in 'a' as well, ie. not just [0:n], but also [1:n], [2:n], etc. I see some functions within scipy.stats.stats, but I'm struggling on figuring out how to put it together. thanks, matt From alex_work at live.de Mon Aug 2 15:35:56 2010 From: alex_work at live.de (Alex Kraus) Date: Mon, 2 Aug 2010 21:35:56 +0200 Subject: [Numpy-discussion] numpy.vectorize fails, howto avoid hardcoding parameters? Message-ID: Hi, I am trying to create a function that calculates the integral of another function. The integral function should later be used in scipy.optimize.leastsq(f, ...), so ideally it should have the format: def f(x, *param) so that it works for a variable number of parameters. While my code works for a fixed number of parameters I cannot get it to work with a variable number of parameters. It seems that numpy.vectorize fails here. Is there a different/better way to do this? from scipy.integrate import quad import numpy as np def integrand_function(x, a, b, c): ??? result = a*x**2 + np.exp(b*x) + np.cos(a*c) ??? return result def define_integral(f, lower, upper): ??? assert(lower < upper) ??? def function(a, b, c): ??????? result = quad(f, lower, upper, args=(a, b, c))[0] ??????? return result ??? return np.vectorize(function) ??? def integrand_function_param(x, *param): ??? a, b, c = param ??? result = a*x**2 + np.exp(b*x) + np.cos(a*c) ??? return result def define_integral_param(f, lower, upper): ??? assert(lower < upper) ??? def function(a, *param): ??????? print(param) ??????? result = quad(f, lower, upper, args=(a, param))[0] ??????? return result ??? return np.vectorize(function) ??? a = np.array([1,2,3,4]) print(integrand_function(1,2,3,4)) # 21.9400368894 f = define_integral(integrand_function, 0.0, 2.0) print(f(a, 1,2)) # [? 8.22342909? 10.41510219? 16.30939667? 16.7647227 ] print(integrand_function_param(1,2,3,4)) # 21.9400368894 fp = define_integral_param(integrand_function_param, 0.0, 2.0) print(fp(a, 1,2)) # ValueError: mismatch between python function inputs and received arguments # fp should later be used in scipy.optimize.leastsq(fp, ... Any help is very appreciated! Alexander From josef.pktd at gmail.com Mon Aug 2 16:57:30 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 2 Aug 2010 16:57:30 -0400 Subject: [Numpy-discussion] numpy.vectorize fails, howto avoid hardcoding parameters? In-Reply-To: References: Message-ID: On Mon, Aug 2, 2010 at 3:35 PM, Alex Kraus wrote: > > Hi, > > I am trying to create a function that calculates the integral of another function. The integral function should later be used in scipy.optimize.leastsq(f, ...), so ideally it should have the format: > > def f(x, *param) > > so that it works for a variable number of parameters. While my code works for a fixed number of parameters I cannot get it to work with a variable number of parameters. It seems that numpy.vectorize fails here. > > Is there a different/better way to do this? > > from scipy.integrate import quad > import numpy as np > > def integrand_function(x, a, b, c): > ??? result = a*x**2 + np.exp(b*x) + np.cos(a*c) > ??? return result > > def define_integral(f, lower, upper): > ??? assert(lower < upper) > > ??? def function(a, b, c): > ??????? result = quad(f, lower, upper, args=(a, b, c))[0] > ??????? return result > ??? return np.vectorize(function) > > def integrand_function_param(x, *param): > ??? a, b, c = param > ??? result = a*x**2 + np.exp(b*x) + np.cos(a*c) > ??? return result > > def define_integral_param(f, lower, upper): > ??? assert(lower < upper) > > ??? def function(a, *param): > ??????? print(param) > ??????? result = quad(f, lower, upper, args=(a, param))[0] > ??????? return result > ??? return np.vectorize(function) > > a = np.array([1,2,3,4]) > > print(integrand_function(1,2,3,4)) > # 21.9400368894 > > f = define_integral(integrand_function, 0.0, 2.0) > print(f(a, 1,2)) > # [? 8.22342909? 10.41510219? 16.30939667? 16.7647227 ] > > print(integrand_function_param(1,2,3,4)) > # 21.9400368894 > > fp = define_integral_param(integrand_function_param, 0.0, 2.0) > print(fp(a, 1,2)) > # ValueError: mismatch between python function inputs and received arguments vectorize cannot infer correctly the number of arguments with *params see http://projects.scipy.org/scipy/ticket/422 the solution that I used in scipy.stats.distributions was to explicitly specify the number of input arguments. nin = ... (I don't remember all the details) Josef > > # fp should later be used in scipy.optimize.leastsq(fp, ... > > Any help is very appreciated! > Alexander > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From david at silveregg.co.jp Mon Aug 2 21:20:19 2010 From: david at silveregg.co.jp (David) Date: Tue, 03 Aug 2010 10:20:19 +0900 Subject: [Numpy-discussion] distutils issue - python 3.1 on windows In-Reply-To: References: Message-ID: <4C576ED3.5040209@silveregg.co.jp> On 08/03/2010 02:57 AM, Pauli Virtanen wrote: > Mon, 02 Aug 2010 12:52:12 -0500, Robert Kern wrote: > [clip] >> I believe we avoided the inspect module because it is quite expensive to >> import. It may not matter inside numpy.distutils, but be wary of >> "fixing" things to use inspect elsewhere. It would be worth extracting >> the commonly-used pieces of inspect (and hacks like this) into an >> internal utility module that is fast to import. > > We actually have `numpy.compat._inspect` and > > from numpy.compat import getargspec > > that could be used here. > Yep, it was added precisely for avoiding the slow import of upstream inspect, cheers, David From guillaume.c.cherel at gmail.com Tue Aug 3 10:12:11 2010 From: guillaume.c.cherel at gmail.com (=?UTF-8?Q?Guillaume_Ch=C3=A9rel?=) Date: Tue, 3 Aug 2010 16:12:11 +0200 Subject: [Numpy-discussion] floating point arithmetic issue In-Reply-To: References: <4C52AD5B.2090108@gmail.com> <4C52AEF5.5070605@silveregg.co.jp> <4C52C3C3.7070509@gmail.com> Message-ID: Hi Pauli, > If your circles are quite small, you probably want to clip the "painting" > to a box not much larger than a single circle: > > # untested, something like below > def point_to_index(x, y, pad=0): > ? ?return np.clip(200 * rr / (xmax - xmin) + pad, 0, 200), \ > ? ? ? ? ? np.clip(200 * rr / (ymax - ymin) + pad, 0, 200) I'm not sure I understand what you're doing with the clip function here. Could you say what are the two returned arrays supposed to represent? Thanks guillaume > > i0, j0 = point_to_index(xx - rr, yy - rr, pad=-2) > i1, j1 = point_to_index(xx + rr, yy + rr, pad=2) > > box = np.index_exp[i0:j0,i1:j1] > mask[box] |= (grid_x[box] - xx)**2 + (grid_y[box] - yy)**2 < rr**2 > # same as: mask[i0:j0,i1:j1] |= (grid_x[i0:j0,i1:j1] ... > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From gammelmark at phys.au.dk Tue Aug 3 10:55:12 2010 From: gammelmark at phys.au.dk (=?ISO-8859-1?Q?S=F8ren_Gammelmark?=) Date: Tue, 03 Aug 2010 16:55:12 +0200 Subject: [Numpy-discussion] Installing numpy with MKL Message-ID: <4C582DD0.4050505@phys.au.dk> Hi everyone I realize that this e-mail has been written a lot of times before. I have, however, not been able to make it work even though I have looked through a lot of e-mails on the mailinglist. I apologize if the solution is written neatly somewhere. As many others I would like to install numpy with MKL support on a cluster. The cluster contains dual core AMD Opteron machines. The reason for doing this is to enable me to perform multi-threaded matrix-operations (matrix-multiplications and singular value decompositions primarily). Additionally I need to integrate some Fortran code also using Intel MKL into numpy using f2py. I think I need to use MKL for this, but correct me if I am wrong. I have tried to follow the guide at http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974. I have compiled my own python and installed it locally using the LD_RUN_PATH suggested there using gcc (I only have access to the intel 9.1 compilers which seems to have a bug preventing me from compiling python with it (see http://bugs.python.org/issue1689617)). This works without any problems. I have access to Intel MKL 10.2.5.035 and have the following site.cfg for numpy 1.4.1: [DEFAULT] library_dirs = /home/sgamark/usr/lib # for python is installed with prefix $HOME/usr include_dirs = /home/sgamark/usr/include [mkl] library_dirs = /com/intel/mkl/10.2.5.035/lib/em64t include_dirs = /com/intel/mkl/10.2.5.035/include lapack_libs = mkl_lapack mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, pthread, mkl_def and have changes numpy/distutils/intelcompiler.py according to http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974 but with cc_exe = 'icc -fPIC -openmp -fomit-frame-pointer -O2 -g'. I have include openmp since I believe it is necessary for using the threading features of MKL (correct me if I am wrong). I have also tried modifiying numpy/distutils/system_info.py as described in http://article.gmane.org/gmane.comp.python.numeric.general/18877/match=intel+mkl but without any effect. However, executing "python setup.py config" shows that the setup-script is unable to find any of the libraries I have (see attached file). If I try to compile anyway MKL is not used and setting OMP_NUM_THREADS have no effect. At this point I am a bit frustrated, since it seems that it should not be too difficult to get to work since it appears that other people have no problems. Indeed, several people I work with have no problem doing this with intel 10.1 compilers, MKL 11.1 on Intel Itanium machines. I am not particularly experienced with this sort of thing but I usually have few problems compiling a stand-alone Fortran or C/C++ program that uses MKL. I would greatly appreciate it if anyone could help with this. Regards S?ren Gammelmark P.s: The operating system is Linux/CentOS 4.4, x86_64 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: setup_conf URL: From sierra_mtnview at sbcglobal.net Tue Aug 3 12:23:40 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Tue, 03 Aug 2010 09:23:40 -0700 Subject: [Numpy-discussion] Changing a matrix element into a scalar Message-ID: <4C58428C.7060404@sbcglobal.net> How do I access 1.2 in such a way as to end up with a float? I keep getting a matrix. from numpy import matrix m = matrix([[1.2],[2.3]]) -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "Republicans are always complaining that government is out of control. If they get into power, they will prove it." -- R. J. Rourke Web Page: From jsseabold at gmail.com Tue Aug 3 12:28:24 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 3 Aug 2010 12:28:24 -0400 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: <4C58428C.7060404@sbcglobal.net> References: <4C58428C.7060404@sbcglobal.net> Message-ID: On Tue, Aug 3, 2010 at 12:23 PM, Wayne Watson wrote: > How do I access 1.2 in such a way as to end up with a float? I keep > getting a matrix. > from numpy import matrix > m = matrix([[1.2],[2.3]]) m[0].item() ? Skipper From PHobson at Geosyntec.com Tue Aug 3 12:28:49 2010 From: PHobson at Geosyntec.com (PHobson at Geosyntec.com) Date: Tue, 3 Aug 2010 12:28:49 -0400 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: <4C58428C.7060404@sbcglobal.net> References: <4C58428C.7060404@sbcglobal.net> Message-ID: Wayne, Matrices are two dimensional arrays so you need two indices to access an individual element: In [1]: from numpy import matrix In [2]: m = matrix([[1.2],[2.3]]) In [3]: m[0,0] Out[3]: 1.2 -paul -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Wayne Watson Sent: Tuesday, August 03, 2010 9:24 AM To: Discussion of Numerical Python Subject: [Numpy-discussion] Changing a matrix element into a scalar How do I access 1.2 in such a way as to end up with a float? I keep getting a matrix. from numpy import matrix m = matrix([[1.2],[2.3]]) -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "Republicans are always complaining that government is out of control. If they get into power, they will prove it." -- R. J. Rourke Web Page: _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From dagss at student.matnat.uio.no Tue Aug 3 12:33:52 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Aug 2010 18:33:52 +0200 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <4C582DD0.4050505@phys.au.dk> References: <4C582DD0.4050505@phys.au.dk> Message-ID: <4C5844F0.10907@student.matnat.uio.no> S?ren Gammelmark wrote: > Hi everyone > > I realize that this e-mail has been written a lot of times before. I > have, however, not been able to make it work even though I have looked > through a lot of e-mails on the mailinglist. I apologize if the > solution is written neatly somewhere. In my experience, it is not. I got this working, my config etc. is in this thread (site.cfg is in my second post): http://www.mail-archive.com/numpy-discussion at scipy.org/msg23151.html If you try that config and post the exact error you get if it does not work, perhaps we'll get somewhere... Dag Sverre From alan.isaac at gmail.com Tue Aug 3 12:56:13 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 03 Aug 2010 12:56:13 -0400 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: <4C58428C.7060404@sbcglobal.net> References: <4C58428C.7060404@sbcglobal.net> Message-ID: <4C584A2D.5070600@gmail.com> On 8/3/2010 12:23 PM, Wayne Watson wrote: > How do I access 1.2 in such a way as to end up with a float? I keep > getting a matrix. > from numpy import matrix > m = matrix([[1.2],[2.3]]) Matrices have the odd (and imo undesirable) property that m[0,0] != m[0][0] You want the former. For a discussion that eventually aborted, see http://www.scipy.org/MatrixIndexing Alan Isaac From sierra_mtnview at sbcglobal.net Tue Aug 3 12:59:52 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Tue, 03 Aug 2010 09:59:52 -0700 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: References: <4C58428C.7060404@sbcglobal.net> Message-ID: <4C584B08.9030803@sbcglobal.net> Thank you. That's almost amusing. Too logical for Python. I fully expected something like m{0}[0](0)[0:0]. :-) I don't think the tentative Numpy tutorial mentions it. On 8/3/2010 9:28 AM, PHobson at Geosyntec.com wrote: > Wayne, > > Matrices are two dimensional arrays so you need two indices to access an individual element: > > In [1]: from numpy import matrix > > In [2]: m = matrix([[1.2],[2.3]]) > > In [3]: m[0,0] > Out[3]: 1.2 > > -paul > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Wayne Watson > Sent: Tuesday, August 03, 2010 9:24 AM > To: Discussion of Numerical Python > Subject: [Numpy-discussion] Changing a matrix element into a scalar > > How do I access 1.2 in such a way as to end up with a float? I keep > getting a matrix. > from numpy import matrix > m = matrix([[1.2],[2.3]]) > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet "Republicans are always complaining that government is out of control. If they get into power, they will prove it." -- R. J. Rourke Web Page: From josh.holbrook at gmail.com Tue Aug 3 13:29:21 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Tue, 3 Aug 2010 09:29:21 -0800 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: <4C584B08.9030803@sbcglobal.net> References: <4C58428C.7060404@sbcglobal.net> <4C584B08.9030803@sbcglobal.net> Message-ID: I don't think I've ever actually seen someone use the matrix datatype instead of the array datatype. Hopefully nobody minds me asking the noob question: What's the advantage of the matrix datatype? --Josh On Tue, Aug 3, 2010 at 8:59 AM, Wayne Watson wrote: > Thank you. That's ?almost amusing. Too logical for Python. I fully > expected something like m{0}[0](0)[0:0]. :-) > > I don't think the tentative Numpy tutorial mentions it. > > On 8/3/2010 9:28 AM, PHobson at Geosyntec.com wrote: >> Wayne, >> >> Matrices are two dimensional arrays so you need two indices to access an individual element: >> >> In [1]: from numpy import matrix >> >> In [2]: m = matrix([[1.2],[2.3]]) >> >> In [3]: m[0,0] >> Out[3]: 1.2 >> >> -paul >> >> -----Original Message----- >> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Wayne Watson >> Sent: Tuesday, August 03, 2010 9:24 AM >> To: Discussion of Numerical Python >> Subject: [Numpy-discussion] Changing a matrix element into a scalar >> >> How do I access 1.2 in such a way as to end up with a float? I keep >> getting a matrix. >> from numpy import matrix >> m = matrix([[1.2],[2.3]]) >> >> > > -- > ? ? ? ? ? ?Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? ?(121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ? Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ?"Republicans are always complaining that government is > ? ? ? ? ? ? ? out of control. If they get into power, they will > ? ? ? ? ? ? ? prove it." -- R. J. Rourke > > > ? ? ? ? ? ? ? ? ? ? Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From alan.isaac at gmail.com Tue Aug 3 14:44:07 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 03 Aug 2010 14:44:07 -0400 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: References: <4C58428C.7060404@sbcglobal.net> <4C584B08.9030803@sbcglobal.net> Message-ID: <4C586377.4070408@gmail.com> On 8/3/2010 1:29 PM, Joshua Holbrook wrote: > What's the > advantage of the matrix datatype? As it turns out, that's a controversial question. ;-) One answer: pedagogy (for those used to matrices). A related answer: succinctness and readability of *some* code. >>> a,b,c = np.array([[1,2,3]]), np.array([[4],[5],[6]]), np.arange(9).reshape((3,3)) >>> d = np.dot(b,np.dot(a,c)) # matrix multiplication >>> d2 = b.dot(a.dot(c)) # NumPy 1.5+ >>> d5 = d.dot(d).dot(d).dot(d).dot(d) #matrix exponentiation >>> d[0,0] = 0 >>> di = np.linalg.inv(d) #matix inverse >>> A,B,C = np.mat('1 2 3'), np.mat('4;5;6'), np.mat(np.arange(9).reshape((3,3))) >>> D = B * (A * C) #matrix multiplication >>> D5 = D**5 #matrix exponentiation >>> D[0,0] = 0 >>> DI = D.I #matix inverse fwiw, Alan Isaac From josh.holbrook at gmail.com Tue Aug 3 14:51:01 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Tue, 3 Aug 2010 10:51:01 -0800 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: <4C586377.4070408@gmail.com> References: <4C58428C.7060404@sbcglobal.net> <4C584B08.9030803@sbcglobal.net> <4C586377.4070408@gmail.com> Message-ID: On Tue, Aug 3, 2010 at 10:44 AM, Alan G Isaac wrote: > On 8/3/2010 1:29 PM, Joshua Holbrook wrote: >> What's the >> advantage of the matrix datatype? > > > As it turns out, that's a controversial question. ?;-) > > One answer: pedagogy (for those used to matrices). > > A related answer: succinctness and readability of *some* code. > > ? ? ? ? >>> a,b,c = np.array([[1,2,3]]), np.array([[4],[5],[6]]), np.arange(9).reshape((3,3)) > ? ? ? ? >>> d = np.dot(b,np.dot(a,c)) ? # matrix multiplication > ? ? ? ? >>> d2 = b.dot(a.dot(c)) ? ? ? ?# NumPy 1.5+ > ? ? ? ? >>> d5 = d.dot(d).dot(d).dot(d).dot(d) ?#matrix exponentiation > ? ? ? ? >>> d[0,0] = 0 > ? ? ? ? >>> di = np.linalg.inv(d) ? ? ? #matix inverse > ? ? ? ? >>> A,B,C = np.mat('1 2 3'), np.mat('4;5;6'), np.mat(np.arange(9).reshape((3,3))) > ? ? ? ? >>> D = B * (A * C) ? ? ? ? ? ? #matrix multiplication > ? ? ? ? >>> D5 = D**5 ? ? ? ? ? ? ? ? ? #matrix exponentiation > ? ? ? ? >>> D[0,0] = 0 > ? ? ? ? >>> DI = D.I ? ? ? ? ? ? ? ? ? ?#matix inverse > > fwiw, > Alan Isaac > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I see. Thanks Alan! --Josh From Chris.Barker at noaa.gov Tue Aug 3 15:10:46 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 03 Aug 2010 12:10:46 -0700 Subject: [Numpy-discussion] Changing a matrix element into a scalar In-Reply-To: References: <4C58428C.7060404@sbcglobal.net> <4C584B08.9030803@sbcglobal.net> Message-ID: <4C5869B6.8070006@noaa.gov> Joshua Holbrook wrote: > I don't think I've ever actually seen someone use the matrix datatype > instead of the array datatype. > > Hopefully nobody minds me asking the noob question: What's the > advantage of the matrix datatype? The advantage of the matrix datatype is that it is a matrix in the linear algebra sense. i.e. A*B is matrix multiplication. As a matrix is, by definition, a 2-d array, matrix operations return 2-d arrays -- hence the OP's issue. I think many of us find n-d arrays far more powerful and useful, and the extra overhead of code for the handful of linear algebra operations in a given piece of code is well worth it. For those that really want a natural way to express linear algebra, the matrix class really needs some more work to do the job well. (see wiki and discussion on this list). However, no one has stepped up to do the hard work of making any of those suggestions a reality, so it's not been done. -Chris > --Josh > > On Tue, Aug 3, 2010 at 8:59 AM, Wayne Watson > wrote: >> Thank you. That's almost amusing. Too logical for Python. I fully >> expected something like m{0}[0](0)[0:0]. :-) >> >> I don't think the tentative Numpy tutorial mentions it. >> >> On 8/3/2010 9:28 AM, PHobson at Geosyntec.com wrote: >>> Wayne, >>> >>> Matrices are two dimensional arrays so you need two indices to access an individual element: >>> >>> In [1]: from numpy import matrix >>> >>> In [2]: m = matrix([[1.2],[2.3]]) >>> >>> In [3]: m[0,0] >>> Out[3]: 1.2 >>> >>> -paul >>> >>> -----Original Message----- >>> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Wayne Watson >>> Sent: Tuesday, August 03, 2010 9:24 AM >>> To: Discussion of Numerical Python >>> Subject: [Numpy-discussion] Changing a matrix element into a scalar >>> >>> How do I access 1.2 in such a way as to end up with a float? I keep >>> getting a matrix. >>> from numpy import matrix >>> m = matrix([[1.2],[2.3]]) >>> >>> >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> "Republicans are always complaining that government is >> out of control. If they get into power, they will >> prove it." -- R. J. Rourke >> >> >> Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dwf at cs.toronto.edu Tue Aug 3 15:52:55 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Tue, 3 Aug 2010 15:52:55 -0400 Subject: [Numpy-discussion] NumPy segfault when running tests (2.7, 1.4.1) Message-ID: <78CC6B1B-590D-4BF8-BC12-AD4621B0F41D@cs.toronto.edu> This was using the Intel MKL/icc to compile NumPy and also icc to compile Python on a shared cluster, but I don't think that's relevant given where the segmentation fault occurs... gpc-f104n084-$ gdb python GNU gdb Fedora (6.8-27.el5) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... (gdb) (gdb) run Starting program: /home/dwf/pkg/bin/python [Thread debugging using libthread_db enabled] Python 2.7 (r27:82500, Aug 3 2010, 15:33:51) [GCC Intel(R) C++ gcc 4.1 mode] on linux2 Type "help", "copyright", "credits" or "license" for more information. [New Thread 0x2b76149111a0 (LWP 23034)] >>> import numpy numpy>>> numpy.test() Running unit tests for numpy NumPy version 1.4.1 NumPy is installed in /home/dwf/pkg/lib/python2.7/site-packages/numpy Python version 2.7 (r27:82500, Aug 3 2010, 15:33:51) [GCC Intel(R) C++ gcc 4.1 mode] nose version 0.11.4 ................................................................................................................................... Program received signal SIGSEGV, Segmentation fault. 0x0000000000500916 in PyErr_WarnEx (category=0x11eb6c54, text=0x5f90c0 "PyOS_ascii_strtod and PyOS_ascii_atof are deprecated. Use PyOS_string_to_double instead.", stack_level=0) at Python/_warnings.c:719 719 res = do_warn(message, category, stack_level); (gdb) From pav at iki.fi Tue Aug 3 16:09:13 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 3 Aug 2010 20:09:13 +0000 (UTC) Subject: [Numpy-discussion] NumPy segfault when running tests (2.7, 1.4.1) References: <78CC6B1B-590D-4BF8-BC12-AD4621B0F41D@cs.toronto.edu> Message-ID: Tue, 03 Aug 2010 15:52:55 -0400, David Warde-Farley wrote: [clip] > in PyErr_WarnEx (category=0x11eb6c54, > text=0x5f90c0 "PyOS_ascii_strtod and PyOS_ascii_atof are deprecated. > Use PyOS_string_to_double instead.", stack_level=0) at > Python/_warnings.c:719 > 719 res = do_warn(message, category, stack_level); > (gdb) That was probably fixed in r8394 in trunk. But to be sure, can you supply the whole stack trace (type "bt" in the gdb prompt). -- Pauli Virtanen From mkraning at gmail.com Tue Aug 3 16:16:54 2010 From: mkraning at gmail.com (matt_in_nyc) Date: Tue, 3 Aug 2010 13:16:54 -0700 (PDT) Subject: [Numpy-discussion] Tensor-Like Outer Product Question Message-ID: <29339226.post@talk.nabble.com> I am trying to perform the following operation: X is an m by n matrix, and I want to store outer products of the form Y[i] = numpy.outer(X[i,:], X[i,:]), leading to the relation Y[i,j,k] = X[i,j]*X[i,k] for i = 0,...,m-1; j,k = 0,...,n-1. I am trying to think of how to do this using tensordot, but so far I am finding no inspiration. Some far, my only solution has been to loop over i Y = numpy.empty([m,n,n]) for i in range(m): Y[i] = numpy.outer(X[i,:], X[i,:]) but this is fairly slow as for my dataset, m is of order 10^7 or 10^8 and n is around 20. Any help on how to vectorize/tensorize this operation to avoid the for loop would be much appreciated. -- View this message in context: http://old.nabble.com/Tensor-Like-Outer-Product-Question-tp29339226p29339226.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From gammelmark at phys.au.dk Tue Aug 3 16:31:41 2010 From: gammelmark at phys.au.dk (=?ISO-8859-1?Q?S=F8ren_Gammelmark?=) Date: Tue, 03 Aug 2010 22:31:41 +0200 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <4C5844F0.10907@student.matnat.uio.no> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> Message-ID: <4C587CAD.4040204@phys.au.dk> I feel a bit silly :-) The compile now appear to work. Sort of. I discovered I had sneaken in a "pthreads" in the mkl_libs line, which is obviously not in the mkl-directory (I had stolen this line from a stand-alone program I wrote at some point). The error message is a bit confusing though - the python setup.py-script seems to claim that none of the libraries are there. In any case the compile works now. However, when I try to run the python tests, python crashes because it cannot find libmkl_def.so. This is weird because LD_LIBRARY_PATH is set correctly. I also tried compiling with LD_RUN_PATH=$MKLROOT/lib/em64t but to no avail. I then tried adding mkl_def to the mkl_libs-line and now it works. However, I seem to recall reading somewhere that this is not a good idea. I guess that this is related to what your referred post was about? Did you end up using the patch or have you found a less invasive method since? Now on to SciPy... :-) Thankyou for your help S?ren On 03-08-2010 18:33, Dag Sverre Seljebotn wrote: > S?ren Gammelmark wrote: > >> Hi everyone >> >> I realize that this e-mail has been written a lot of times before. I >> have, however, not been able to make it work even though I have looked >> through a lot of e-mails on the mailinglist. I apologize if the >> solution is written neatly somewhere. >> > In my experience, it is not. > > I got this working, my config etc. is in this thread (site.cfg is in my > second post): > > http://www.mail-archive.com/numpy-discussion at scipy.org/msg23151.html > > If you try that config and post the exact error you get if it does not > work, perhaps we'll get somewhere... > > Dag Sverre > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From friedrichromstedt at gmail.com Tue Aug 3 16:42:07 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Tue, 3 Aug 2010 22:42:07 +0200 Subject: [Numpy-discussion] Tensor-Like Outer Product Question In-Reply-To: <29339226.post@talk.nabble.com> References: <29339226.post@talk.nabble.com> Message-ID: 2010/8/3 matt_in_nyc : > X is an m by n matrix, and I want to store outer products of the form Y[i] = > numpy.outer(X[i,:], X[i,:]), leading to the relation Y[i,j,k] = > X[i,j]*X[i,k] for i = 0,...,m-1; j,k = 0,...,n-1. ?I am trying to think of > how to do this using tensordot, but so far I am finding no inspiration. What about: X[:, :, numpy.newaxis] * X[:, numpy.newaxis, :] ? untested. Friedrich From bsouthey at gmail.com Tue Aug 3 17:10:47 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 03 Aug 2010 16:10:47 -0500 Subject: [Numpy-discussion] Python 3.2 crashes numpy with undefined symbol PyCObject_AsVoidPtr Message-ID: <4C5885D7.7040005@gmail.com> Hi, Since I was testing the 1.5 beta, I also tested the alpha release of Python 3.2 on Linux 64-bit (gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)). While my other Python versions passed the tests (once I copied the necessary tools directory for Python3.1), Python3.2 has problems so I created ticket 1571: http://projects.scipy.org/numpy/ticket/1571 The minor change is that string.maketrans function, depreciated in Python 3.1, has been removed. http://docs.python.org/dev/py3k/whatsnew/3.2.html However the more serious problem is that both 1.5 beta and the current svn version crashed: $ python3.2 -c "import numpy; numpy.test()" Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.2/site-packages/numpy/__init__.py", line 136, in from . import add_newdocs File "/usr/local/lib/python3.2/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/usr/local/lib/python3.2/site-packages/numpy/lib/__init__.py", line 4, in from .type_check import * File "/usr/local/lib/python3.2/site-packages/numpy/lib/type_check.py", line 8, in import numpy.core.numeric as _nx File "/usr/local/lib/python3.2/site-packages/numpy/core/__init__.py", line 5, in from . import multiarray ImportError: /usr/local/lib/python3.2/site-packages/numpy/core/multiarray.so: undefined symbol: PyCObject_AsVoidPtr This warning that does not appears in Python3.1: In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:28: numpy/core/src/multiarray/convert_datatype.c: In function ?PyArray_GetCastFunc?: numpy/core/src/multiarray/convert_datatype.c:106: warning: implicit declaration of function ?PyCObject_AsVoidPtr? numpy/core/src/multiarray/convert_datatype.c:106: warning: assignment makes pointer from integer without a cast Bruce From robert.faryabi at gmail.com Tue Aug 3 17:40:13 2010 From: robert.faryabi at gmail.com (Robert Faryabi) Date: Tue, 3 Aug 2010 17:40:13 -0400 Subject: [Numpy-discussion] Tabular: Importing data from file Message-ID: Hi there; I'm using Tabular Package for manipulating tab-delimited data. There is a small problem that I cannot get my head around it. When I construct my tabarray from file, the blank fields are replaced by "nan". Does any one knows how to just keep them as empty string (ie. ' ')? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Tue Aug 3 17:46:07 2010 From: faltet at pytables.org (Francesc Alted) Date: Tue, 3 Aug 2010 22:46:07 +0100 Subject: [Numpy-discussion] Numexpr: erfc function In-Reply-To: <4C58553E.2060407@gmx.de> References: <4C58553E.2060407@gmx.de> Message-ID: Hi Till, 2010/8/3, Till Stensitzki : > Hello, > I am sorry for contacting you directly, i tried but i didn't found > something like a numexpr mailinglist. > Fist of all, thanks for this modul, it really helps to speed up some of > my fitting rounties! > Now for my qustion: Is there any way to add the complementary error > function (erfc in scipy and c) to the supported functions? Yes, it is possible to add more functions to Numexpr indeed. Just start looking at how support for some POSIX functions have been added in file ``missing_posix_functions.inc``, and try to replicate the support code for, for example, `log1p()`. It is not complicated, just time-consuming. If you end with the suport, just send me the patch and I'll add it to the repo. Regarding the mailing list, I don't really think Numexpr would need one for now. I suppose that discussion for Numexpr is welcome in NumPy/SciPy list (and the PyTables list can certainly also be used). Luck! -- Francesc Alted From alan.isaac at gmail.com Tue Aug 3 18:25:30 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 03 Aug 2010 18:25:30 -0400 Subject: [Numpy-discussion] Tabular: Importing data from file In-Reply-To: References: Message-ID: <4C58975A.80009@gmail.com> On 8/3/2010 5:40 PM, Robert Faryabi wrote: > I'm using Tabular Package for manipulating tab-delimited data. > There is a small problem that I cannot get my head around it. > > When I construct my tabarray from file, the blank fields are replaced by "nan". > Does any one knows how to just keep them as empty string (ie. ' ')? My understanding is that ``tabular`` uses NumPy arrays (e.g., float arrays). So nan is natural for missing data. If you are concerned about writing out data as a table, substituting for nans, you could try econpy's table.py:: http://code.google.com/p/econpy/source/browse/trunk/utilities/table.py (See the labeler example in the module docstring.) fwiw, Alan Isaac From pav at iki.fi Tue Aug 3 18:25:50 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 3 Aug 2010 22:25:50 +0000 (UTC) Subject: [Numpy-discussion] distutils issue - python 3.1 on windows References: Message-ID: Mon, 02 Aug 2010 23:48:52 +0800, Ralf Gommers wrote: > I'm trying to get building to work with Python 3.1 under Wine on OS X. > The first thing you run into is a python distutils problem, which is > fixed by replacing line 379 of cygwinccompiler.py with > result = RE_VERSION.search(str(out_string)) Ah yep, cygwinccompiler.py is in Python's distutils, I wondered for a while where the bug was :) But actually, I'm now looking at the cygwinccompiler.py source code, and to me it seems it should be ok. The RE_VERSION is a byte-regular expression, so it's not obvious why making the input a str would help? > File > "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils \mingw32ccompiler.py", > line 177, in link > func(*args[:func.__func__.__code__.co_argcount]) > AttributeError: 'function' object has no attribute '__func__' I fixed this in trunk r8595 (cherry-pick to 1.5.x if you find it works). Pauli From samtygier at yahoo.co.uk Tue Aug 3 19:03:46 2010 From: samtygier at yahoo.co.uk (sam tygier) Date: Wed, 04 Aug 2010 00:03:46 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: On 01/08/10 17:38, Ralf Gommers wrote: > I am pleased to announce the availability of the first beta of NumPy > 1.5.0. This will be the first NumPy release to include support for > Python 3, as well as for Python 2.7. Please try this beta and report any > problems on the NumPy mailing list. the deprecate_old_order.diff patch from ticket #1431 seems not to have been applied yet. this is needed to warn when accessing items of a recarray is going to give them in the wrong order. this will allow the issue to be fixed in a future version. thanks sam From ben.root at ou.edu Tue Aug 3 19:19:27 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 3 Aug 2010 18:19:27 -0500 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <4C587CAD.4040204@phys.au.dk> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> Message-ID: On Tue, Aug 3, 2010 at 3:31 PM, S?ren Gammelmark wrote: > I feel a bit silly :-) > The compile now appear to work. Sort of. I discovered I had sneaken in a > "pthreads" in the mkl_libs line, which is obviously not in the > mkl-directory (I had stolen this line from a stand-alone program I wrote > at some point). The error message is a bit confusing though - the python > setup.py-script seems to claim that none of the libraries are there. > > In any case the compile works now. However, when I try to run the python > tests, python crashes because it cannot find libmkl_def.so. This is > weird because LD_LIBRARY_PATH is set correctly. I also tried compiling > with LD_RUN_PATH=$MKLROOT/lib/em64t but to no avail. I then tried adding > mkl_def to the mkl_libs-line and now it works. However, I seem to recall > reading somewhere that this is not a good idea. I guess that this is > related to what your referred post was about? Did you end up using the > patch or have you found a less invasive method since? > > Now on to SciPy... :-) > > Thankyou for your help > S?ren > > I wouldn't know for sure, but could this be related to changes to the gcc compiler in Fedora 13 (with respect to implicit DSO linking) or would that only be an issue at build-time? http://fedoraproject.org/w/index.php?title=UnderstandingDSOLinkChange Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From gammelmark at phys.au.dk Wed Aug 4 02:09:16 2010 From: gammelmark at phys.au.dk (=?ISO-8859-1?Q?S=F8ren_Gammelmark?=) Date: Wed, 04 Aug 2010 08:09:16 +0200 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> Message-ID: <4C59040C.2020901@phys.au.dk> > > I wouldn't know for sure, but could this be related to changes to the > gcc compiler in Fedora 13 (with respect to implicit DSO linking) or > would that only be an issue at build-time? > > http://fedoraproject.org/w/index.php?title=UnderstandingDSOLinkChange I'm not entirely sure I understand the link, but if it has anything to do with the compiler it seems to me that it should be the Intel compiler. The python I use is compiled with GCC but everything in numpy is done with the Intel compilers. Shouldn't it then be something with the Intel compilers? /S?ren From matthieu.brucher at gmail.com Wed Aug 4 02:18:34 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 4 Aug 2010 08:18:34 +0200 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <4C59040C.2020901@phys.au.dk> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> <4C59040C.2020901@phys.au.dk> Message-ID: 2010/8/4 S?ren Gammelmark : > >> >> I wouldn't know for sure, but could this be related to changes to the >> gcc compiler in Fedora 13 (with respect to implicit DSO linking) or >> would that only be an issue at build-time? >> >> http://fedoraproject.org/w/index.php?title=UnderstandingDSOLinkChange > > I'm not entirely sure I understand the link, but if it has anything to > do with the compiler it seems to me that it should be the Intel > compiler. The python I use is compiled with GCC but everything in numpy > is done with the Intel compilers. Shouldn't it then be something with > the Intel compilers? > > /S?ren Unfortunately, I think you'll ahve to use Dag's patch. MKL has a specific loading procedure since a few releases, you have to abide by it. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From bsouthey at gmail.com Wed Aug 4 09:20:59 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 4 Aug 2010 08:20:59 -0500 Subject: [Numpy-discussion] Python 3.2 crashes numpy with undefined symbol PyCObject_AsVoidPtr In-Reply-To: <4C5885D7.7040005@gmail.com> References: <4C5885D7.7040005@gmail.com> Message-ID: On Tue, Aug 3, 2010 at 4:10 PM, Bruce Southey wrote: > ?Hi, > Since I was testing the 1.5 beta, I also tested the alpha release of Python > 3.2 on Linux 64-bit (gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)). > While my other Python versions passed the tests (once I copied the necessary > tools directory for Python3.1), Python3.2 has problems so I created ticket > 1571: > http://projects.scipy.org/numpy/ticket/1571 > > The minor change is that string.maketrans function, depreciated in Python > 3.1, has been removed. > http://docs.python.org/dev/py3k/whatsnew/3.2.html > > However the more serious problem is that both 1.5 beta and the current svn > version crashed: > > $ python3.2 -c "import numpy; numpy.test()" > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python3.2/site-packages/numpy/__init__.py", line 136, > in > from . import add_newdocs > File "/usr/local/lib/python3.2/site-packages/numpy/add_newdocs.py", line 9, > in > from numpy.lib import add_newdoc > File "/usr/local/lib/python3.2/site-packages/numpy/lib/__init__.py", line 4, > in > from .type_check import * > File "/usr/local/lib/python3.2/site-packages/numpy/lib/type_check.py", line > 8, in > import numpy.core.numeric as _nx > File "/usr/local/lib/python3.2/site-packages/numpy/core/__init__.py", line > 5, in > from . import multiarray > ImportError: > /usr/local/lib/python3.2/site-packages/numpy/core/multiarray.so: undefined > symbol: PyCObject_AsVoidPtr > > > This warning that does not appears in Python3.1: > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:28: > numpy/core/src/multiarray/convert_datatype.c: In function > ?PyArray_GetCastFunc?: > numpy/core/src/multiarray/convert_datatype.c:106: warning: implicit > declaration of function ?PyCObject_AsVoidPtr? > numpy/core/src/multiarray/convert_datatype.c:106: warning: assignment makes > pointer from integer without a cast > > Bruce > Many Thanks for the prompt fixes! Python3.2a1 passes all tests with the svn version 2.0.0.dev8587. Hopefully these can be included in the 1.5 release. Bruce $ python3.2 -c "import numpy; numpy.test()" Running unit tests for numpy NumPy version 2.0.0.dev8587 NumPy is installed in /usr/local/lib/python3.2/site-packages/numpy Python version 3.2a1 (r32a1:83318, Aug 3 2010, 14:17:59) [GCC 4.4.4 20100630 (Red Hat 4.4.4-10)] nose version 3.0.0 ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K......................................................K...........................................................................................................................................................................K............................................................................................K......................K...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Warning: divide by zero encountered in log ....................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 2976 tests in 11.059s OK (KNOWNFAIL=5) From boolegue at gmail.com Wed Aug 4 10:29:33 2010 From: boolegue at gmail.com (Antoine Dechaume) Date: Wed, 4 Aug 2010 16:29:33 +0200 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? Message-ID: Hi, given A=empty([10,10]), I would like to keep A[:5,:5] as a contiguous memory segment. How to do it efficiently? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Aug 4 11:01:16 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 Aug 2010 10:01:16 -0500 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 09:29, Antoine Dechaume wrote: > Hi, > > given A=empty([10,10]), I would like to keep A[:5,:5]?as a contiguous memory > segment. > > How to do it efficiently? I'm not sure I understand what you want. Your Subject line and the body of your email conflict with each other. Can you try to explain what you want in other words? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From boolegue at gmail.com Wed Aug 4 11:17:01 2010 From: boolegue at gmail.com (Antoine Dechaume) Date: Wed, 4 Aug 2010 17:17:01 +0200 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: References: Message-ID: I forgot to refer to resize, sorry about that. A[:5,:5] shows the data I want, but it's not contiguous in memory. A.resize(5,5) is contiguous, but do not contains the data I want. How to get both efficiently? On Wed, Aug 4, 2010 at 5:01 PM, Robert Kern wrote: > On Wed, Aug 4, 2010 at 09:29, Antoine Dechaume wrote: > > Hi, > > > > given A=empty([10,10]), I would like to keep A[:5,:5] as a contiguous > memory > > segment. > > > > How to do it efficiently? > > I'm not sure I understand what you want. Your Subject line and the > body of your email conflict with each other. Can you try to explain > what you want in other words? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Wed Aug 4 11:20:44 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 4 Aug 2010 11:20:44 -0400 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: References: Message-ID: <89F38770-66C5-40FB-BDB5-79B339875647@yale.edu> > A[:5,:5] shows the data I want, but it's not contiguous in memory. > A.resize(5,5) is contiguous, but do not contains the data I want. > > How to get both efficiently? A[:5,:5].copy() will give you a new, contiguous array that has the same contents as A[5:,5:], but in a new chunk of memory. Is this what you need? On Aug 4, 2010, at 11:17 AM, Antoine Dechaume wrote: > I forgot to refer to resize, sorry about that. > > A[:5,:5] shows the data I want, but it's not contiguous in memory. > A.resize(5,5) is contiguous, but do not contains the data I want. > > How to get both efficiently? > > > On Wed, Aug 4, 2010 at 5:01 PM, Robert Kern > wrote: > On Wed, Aug 4, 2010 at 09:29, Antoine Dechaume > wrote: > > Hi, > > > > given A=empty([10,10]), I would like to keep A[:5,:5] as a > contiguous memory > > segment. > > > > How to do it efficiently? > > I'm not sure I understand what you want. Your Subject line and the > body of your email conflict with each other. Can you try to explain > what you want in other words? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From boolegue at gmail.com Wed Aug 4 11:31:05 2010 From: boolegue at gmail.com (Antoine Dechaume) Date: Wed, 4 Aug 2010 17:31:05 +0200 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: <89F38770-66C5-40FB-BDB5-79B339875647@yale.edu> References: <89F38770-66C5-40FB-BDB5-79B339875647@yale.edu> Message-ID: Yes it is, but is there a way to do it in-place? On Wed, Aug 4, 2010 at 5:20 PM, Zachary Pincus wrote: > > A[:5,:5] shows the data I want, but it's not contiguous in memory. > > A.resize(5,5) is contiguous, but do not contains the data I want. > > > > How to get both efficiently? > > A[:5,:5].copy() > will give you a new, contiguous array that has the same contents as > A[5:,5:], but in a new chunk of memory. Is this what you need? > > > > > On Aug 4, 2010, at 11:17 AM, Antoine Dechaume wrote: > > > I forgot to refer to resize, sorry about that. > > > > A[:5,:5] shows the data I want, but it's not contiguous in memory. > > A.resize(5,5) is contiguous, but do not contains the data I want. > > > > How to get both efficiently? > > > > > > On Wed, Aug 4, 2010 at 5:01 PM, Robert Kern > > wrote: > > On Wed, Aug 4, 2010 at 09:29, Antoine Dechaume > > wrote: > > > Hi, > > > > > > given A=empty([10,10]), I would like to keep A[:5,:5] as a > > contiguous memory > > > segment. > > > > > > How to do it efficiently? > > > > I'm not sure I understand what you want. Your Subject line and the > > body of your email conflict with each other. Can you try to explain > > what you want in other words? > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a harmless > > enigma that is made terrible by our own mad attempt to interpret it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Aug 4 11:34:15 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 Aug 2010 23:34:15 +0800 Subject: [Numpy-discussion] distutils issue - python 3.1 on windows In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 6:25 AM, Pauli Virtanen wrote: > Mon, 02 Aug 2010 23:48:52 +0800, Ralf Gommers wrote: > > I'm trying to get building to work with Python 3.1 under Wine on OS X. > > The first thing you run into is a python distutils problem, which is > > fixed by replacing line 379 of cygwinccompiler.py with > > result = RE_VERSION.search(str(out_string)) > > Ah yep, cygwinccompiler.py is in Python's distutils, I wondered for a > while where the bug was :) > > But actually, I'm now looking at the cygwinccompiler.py source code, and > to me it seems it should be ok. The RE_VERSION is a byte-regular > expression, so it's not obvious why making the input a str would help? > > I haven't started using py3k yet so I'm still a bit fuzzy about bytes vs string. But it's easy to try in the interpreter: >>> import re >>> RE_VERSION = re.compile('(\d+\.\d+(\.\d+)*)') >>> cmd = "gcc -dumpversion" >>> from subprocess import Popen, PIPE >>> out = Popen(cmd, shell=True, stdout=PIPE).stdout >>> out_string = out.read() >>> out.close() >>> result = RE_VERSION.search(out_string) Traceback (most recent call last): File "", line 1, in TypeError: can't use a string pattern on a bytes-like object > > File > > "Z:\Users\rgommers\Code\numpy\build\py3k\numpy\distutils > \mingw32ccompiler.py", > > line 177, in link > > func(*args[:func.__func__.__code__.co_argcount]) > > AttributeError: 'function' object has no attribute '__func__' > > I fixed this in trunk r8595 (cherry-pick to 1.5.x if you find it works). > That works, thanks. Then there's was still one more problem, _dotblas.c was not py3k ready. With this fix I can build everything under Wine: http://github.com/rgommers/numpy/tree/build-py3k Does that look fine? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Aug 4 11:36:08 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 Aug 2010 23:36:08 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 7:03 AM, sam tygier wrote: > On 01/08/10 17:38, Ralf Gommers wrote: > > I am pleased to announce the availability of the first beta of NumPy > > 1.5.0. This will be the first NumPy release to include support for > > Python 3, as well as for Python 2.7. Please try this beta and report any > > problems on the NumPy mailing list. > > the deprecate_old_order.diff patch from ticket #1431 seems not to have been > applied yet. > > this is needed to warn when accessing items of a recarray is going to give > them in the wrong order. this will allow the issue to be fixed in a future > version. > > Thanks for pointing that out. I'll check all deprecations before the next beta/rc. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Aug 4 11:40:53 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 Aug 2010 10:40:53 -0500 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: References: <89F38770-66C5-40FB-BDB5-79B339875647@yale.edu> Message-ID: On Wed, Aug 4, 2010 at 10:31, Antoine Dechaume wrote: > Yes it is, but is there a way to do it in-place? No. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From zachary.pincus at yale.edu Wed Aug 4 11:52:28 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 4 Aug 2010 11:52:28 -0400 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: References: <89F38770-66C5-40FB-BDB5-79B339875647@yale.edu> Message-ID: <35518F81-3AE8-404B-BAF2-5D094FEC5225@yale.edu> > Yes it is, but is there a way to do it in-place? So you want the first 25 elements of the array (in a flat "contiguous" view) to contain the 25 elements of A[:5,:5]? This will do that, but having to do stuff like this (rather than just copying the memory region) might be indicative that maybe your code design might not really be right. (Why does it absolutely have to be in-place? Memory pressure?) In [34]: a = numpy.array(numpy.arange(100, dtype=int).reshape(10,10)) In [35]: a Out[35]: array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], [20, 21, 22, 23, 24, 25, 26, 27, 28, 29], [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], [40, 41, 42, 43, 44, 45, 46, 47, 48, 49], [50, 51, 52, 53, 54, 55, 56, 57, 58, 59], [60, 61, 62, 63, 64, 65, 66, 67, 68, 69], [70, 71, 72, 73, 74, 75, 76, 77, 78, 79], [80, 81, 82, 83, 84, 85, 86, 87, 88, 89], [90, 91, 92, 93, 94, 95, 96, 97, 98, 99]]) In [36]: a[:5,:5] Out[36]: array([[ 0, 1, 2, 3, 4], [10, 11, 12, 13, 14], [20, 21, 22, 23, 24], [30, 31, 32, 33, 34], [40, 41, 42, 43, 44]]) In [37]: a.flat[:25] = a[:5,:5] In [38]: a.resize((5,5)) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/zpincus/ in () ValueError: cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function In [39]: b = numpy.ndarray(buffer=a, shape=(5,5), dtype=int) In [40]: b Out[40]: array([[ 0, 1, 2, 3, 4], [10, 11, 12, 13, 14], [20, 21, 22, 23, 24], [30, 31, 32, 33, 34], [40, 41, 42, 43, 44]]) In [41]: b.flags.c_contiguous Out[41]: True In [42]: b.flags.owndata Out[42]: False Zach > On Wed, Aug 4, 2010 at 5:20 PM, Zachary Pincus > wrote: > > A[:5,:5] shows the data I want, but it's not contiguous in memory. > > A.resize(5,5) is contiguous, but do not contains the data I want. > > > > How to get both efficiently? > > A[:5,:5].copy() > will give you a new, contiguous array that has the same contents as > A[5:,5:], but in a new chunk of memory. Is this what you need? > > > > > On Aug 4, 2010, at 11:17 AM, Antoine Dechaume wrote: > > > I forgot to refer to resize, sorry about that. > > > > A[:5,:5] shows the data I want, but it's not contiguous in memory. > > A.resize(5,5) is contiguous, but do not contains the data I want. > > > > How to get both efficiently? > > > > > > On Wed, Aug 4, 2010 at 5:01 PM, Robert Kern > > wrote: > > On Wed, Aug 4, 2010 at 09:29, Antoine Dechaume > > wrote: > > > Hi, > > > > > > given A=empty([10,10]), I would like to keep A[:5,:5] as a > > contiguous memory > > > segment. > > > > > > How to do it efficiently? > > > > I'm not sure I understand what you want. Your Subject line and the > > body of your email conflict with each other. Can you try to explain > > what you want in other words? > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a > harmless > > enigma that is made terrible by our own mad attempt to interpret > it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From zachary.pincus at yale.edu Wed Aug 4 11:56:32 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 4 Aug 2010 11:56:32 -0400 Subject: [Numpy-discussion] ndarray.resize that preserve view content ? In-Reply-To: <35518F81-3AE8-404B-BAF2-5D094FEC5225@yale.edu> References: <89F38770-66C5-40FB-BDB5-79B339875647@yale.edu> <35518F81-3AE8-404B-BAF2-5D094FEC5225@yale.edu> Message-ID: <614455AC-F7BA-486C-AF1C-D6539CE41331@yale.edu> Oh and PS. Robert's right that there's no general way to do this! What I have only works because the data existing in the first 25 elements of A that get clobbered by the copy operation aren't the same data that are being copied (or where they are, the new copy is identical to the old one). Other slices won't have this property... A[:] = A[::-1] e.g. will fail totally. On Aug 4, 2010, at 11:52 AM, Zachary Pincus wrote: >> Yes it is, but is there a way to do it in-place? > > So you want the first 25 elements of the array (in a flat "contiguous" > view) to contain the 25 elements of A[:5,:5]? This will do that, but > having to do stuff like this (rather than just copying the memory > region) might be indicative that maybe your code design might not > really be right. (Why does it absolutely have to be in-place? Memory > pressure?) > > In [34]: a = numpy.array(numpy.arange(100, dtype=int).reshape(10,10)) > > In [35]: a > Out[35]: > array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], > [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], > [20, 21, 22, 23, 24, 25, 26, 27, 28, 29], > [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], > [40, 41, 42, 43, 44, 45, 46, 47, 48, 49], > [50, 51, 52, 53, 54, 55, 56, 57, 58, 59], > [60, 61, 62, 63, 64, 65, 66, 67, 68, 69], > [70, 71, 72, 73, 74, 75, 76, 77, 78, 79], > [80, 81, 82, 83, 84, 85, 86, 87, 88, 89], > [90, 91, 92, 93, 94, 95, 96, 97, 98, 99]]) > > In [36]: a[:5,:5] > Out[36]: > array([[ 0, 1, 2, 3, 4], > [10, 11, 12, 13, 14], > [20, 21, 22, 23, 24], > [30, 31, 32, 33, 34], > [40, 41, 42, 43, 44]]) > > In [37]: a.flat[:25] = a[:5,:5] > > In [38]: a.resize((5,5)) > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > > /Users/zpincus/ in () > > ValueError: cannot resize an array that has been referenced or is > referencing > another array in this way. Use the resize function > > In [39]: b = numpy.ndarray(buffer=a, shape=(5,5), dtype=int) > In [40]: b > > Out[40]: > array([[ 0, 1, 2, 3, 4], > [10, 11, 12, 13, 14], > [20, 21, 22, 23, 24], > [30, 31, 32, 33, 34], > [40, 41, 42, 43, 44]]) > > > In [41]: b.flags.c_contiguous > Out[41]: True > > In [42]: b.flags.owndata > Out[42]: False > > > Zach > > >> On Wed, Aug 4, 2010 at 5:20 PM, Zachary Pincus >> wrote: >>> A[:5,:5] shows the data I want, but it's not contiguous in memory. >>> A.resize(5,5) is contiguous, but do not contains the data I want. >>> >>> How to get both efficiently? >> >> A[:5,:5].copy() >> will give you a new, contiguous array that has the same contents as >> A[5:,5:], but in a new chunk of memory. Is this what you need? >> >> >> >> >> On Aug 4, 2010, at 11:17 AM, Antoine Dechaume wrote: >> >>> I forgot to refer to resize, sorry about that. >>> >>> A[:5,:5] shows the data I want, but it's not contiguous in memory. >>> A.resize(5,5) is contiguous, but do not contains the data I want. >>> >>> How to get both efficiently? >>> >>> >>> On Wed, Aug 4, 2010 at 5:01 PM, Robert Kern >>> wrote: >>> On Wed, Aug 4, 2010 at 09:29, Antoine Dechaume >>> wrote: >>>> Hi, >>>> >>>> given A=empty([10,10]), I would like to keep A[:5,:5] as a >>> contiguous memory >>>> segment. >>>> >>>> How to do it efficiently? >>> >>> I'm not sure I understand what you want. Your Subject line and the >>> body of your email conflict with each other. Can you try to explain >>> what you want in other words? >>> >>> -- >>> Robert Kern >>> >>> "I have come to believe that the whole world is an enigma, a >> harmless >>> enigma that is made terrible by our own mad attempt to interpret >> it as >>> though it had an underlying truth." >>> -- Umberto Eco >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Wed Aug 4 15:40:57 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 4 Aug 2010 19:40:57 +0000 (UTC) Subject: [Numpy-discussion] distutils issue - python 3.1 on windows References: Message-ID: Wed, 04 Aug 2010 23:34:15 +0800, Ralf Gommers wrote: [clip] > I haven't started using py3k yet so I'm still a bit fuzzy about bytes > vs string. But it's easy to try in the interpreter: > >>>> import re >>>> RE_VERSION = re.compile('(\d+\.\d+(\.\d+)*)') In the Python 3.1 version I have, this line reads RE_VERSION = re.compile(b'(\d+\.\d+(\.\d+)*)') which makes it a byte-regular expression. [clip] > That works, thanks. Then there's was still one more problem, _dotblas.c > was not py3k ready. Doh! How did I miss that (answer: no tests, and the import error is masked :) > With this fix I can build everything under Wine: > http://github.com/rgommers/numpy/tree/build-py3k Does that look fine? It compiles, but will probably crash on import :) You need also to return the module object, see e.g. how it's done in numpy/numarray/_capi.c I committed a fixed version. Cheers, Pauli From PHobson at Geosyntec.com Wed Aug 4 19:59:07 2010 From: PHobson at Geosyntec.com (PHobson at Geosyntec.com) Date: Wed, 4 Aug 2010 19:59:07 -0400 Subject: [Numpy-discussion] Quick array value assignment based on common values Message-ID: Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul From jsalvati at u.washington.edu Wed Aug 4 20:33:52 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Wed, 4 Aug 2010 17:33:52 -0700 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: Are they numbered like that? If so you can index into the first array by the second one. x[y[:,0], 1] if you can't get them into an indexable format, I think it's going to be slow no matter how you do it. On Wed, Aug 4, 2010 at 4:59 PM, wrote: > Hey folks, > > I've one array, x, that you could define as follows: > [[1, 2.25], > [2, 2.50], > [3, 2.25], > [4, 0.00], > [8, 0.00], > [9, 2.75]] > > Then my second array, y, is: > [[1, 0.00], > [2, 0.00], > [3, 0.00], > [4, 0.00], > [5, 0.00], > [6, 0.00], > [7, 0.00], > [8, 0.00], > [9, 0.00], > [10,0.00]] > > Is there a concise, Numpythonic way to copy the values of x[:,1] over to > y[:,1] where x[:,0] = y[:,0]? Resulting in, z: > [[1, 2.25], > [2, 2.50], > [3, 2.25], > [4, 0.00], > [5, 0.00], > [6, 0.00], > [7, 0.00], > [8, 0.00], > [9, 2.75], > [10,0.00]] > > My current task has len(x) = 25000 and len(y) = 350000 and looping through > is quite slow unfortunately. > > Many thanks, > -paul > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Wed Aug 4 20:55:28 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 5 Aug 2010 02:55:28 +0200 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: > Is there a concise, Numpythonic way to copy the values of x[:,1] over to > y[:,1] where x[:,0] = y[:,0]? Resulting in, z: First use mask = (x[:,0] == y[:,0]) # integers or mask = np.abs(x[:,0] - y[:,0]) < eps # floats and then y[mask,1] = x[mask,1] Sturla From PHobson at Geosyntec.com Wed Aug 4 20:58:24 2010 From: PHobson at Geosyntec.com (PHobson at Geosyntec.com) Date: Wed, 4 Aug 2010 20:58:24 -0400 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: John, Thanks for the quick reply. Unfortunately, no, they're not indexed like that. The first columns are actually floating-point date numbers from matplotlib.dates.date2num. Looks like this is just going to be painful... Thanks for the tip though. That'll definitely be useful elsewhere. -paul From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of John Salvatier Sent: Wednesday, August 04, 2010 5:34 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Quick array value assignment based on common values Are they numbered like that? If so you can index into the first array by the second one. x[y[:,0], 1] if you can't get them into an indexable format, I think it's going to be slow no matter how you do it. On Wed, Aug 4, 2010 at 4:59 PM, > wrote: Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Wed Aug 4 21:00:37 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 4 Aug 2010 20:00:37 -0500 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 6:59 PM, wrote: > Hey folks, > > I've one array, x, that you could define as follows: > [[1, 2.25], > [2, 2.50], > [3, 2.25], > [4, 0.00], > [8, 0.00], > [9, 2.75]] > > Then my second array, y, is: > [[1, 0.00], > [2, 0.00], > [3, 0.00], > [4, 0.00], > [5, 0.00], > [6, 0.00], > [7, 0.00], > [8, 0.00], > [9, 0.00], > [10,0.00]] > > Is there a concise, Numpythonic way to copy the values of x[:,1] over to > y[:,1] where x[:,0] = y[:,0]? Resulting in, z: > [[1, 2.25], > [2, 2.50], > [3, 2.25], > [4, 0.00], > [5, 0.00], > [6, 0.00], > [7, 0.00], > [8, 0.00], > [9, 2.75], > [10,0.00]] > > My current task has len(x) = 25000 and len(y) = 350000 and looping through > is quite slow unfortunately. > > Many thanks, > -paul > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > My simplest approach would be: y[x[:0]-1] = x # Providing the arrays are nicely ordered and 1st column x is all integer. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Wed Aug 4 21:02:47 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 4 Aug 2010 20:02:47 -0500 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 8:00 PM, G?khan Sever wrote: > > > On Wed, Aug 4, 2010 at 6:59 PM, wrote: > >> Hey folks, >> >> I've one array, x, that you could define as follows: >> [[1, 2.25], >> [2, 2.50], >> [3, 2.25], >> [4, 0.00], >> [8, 0.00], >> [9, 2.75]] >> >> Then my second array, y, is: >> [[1, 0.00], >> [2, 0.00], >> [3, 0.00], >> [4, 0.00], >> [5, 0.00], >> [6, 0.00], >> [7, 0.00], >> [8, 0.00], >> [9, 0.00], >> [10,0.00]] >> >> Is there a concise, Numpythonic way to copy the values of x[:,1] over to >> y[:,1] where x[:,0] = y[:,0]? Resulting in, z: >> [[1, 2.25], >> [2, 2.50], >> [3, 2.25], >> [4, 0.00], >> [5, 0.00], >> [6, 0.00], >> [7, 0.00], >> [8, 0.00], >> [9, 2.75], >> [10,0.00]] >> >> My current task has len(x) = 25000 and len(y) = 350000 and looping through >> is quite slow unfortunately. >> >> Many thanks, >> -paul >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > My simplest approach would be: > > y[x[:0]-1] = x > > # Providing the arrays are nicely ordered and 1st column x is all integer. > > -- > G?khan > With the forgotten comma ;) y[x[:,0]-1] = x -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Wed Aug 4 21:09:23 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Wed, 4 Aug 2010 18:09:23 -0700 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: How exactly are you looping? That sounds absurdly slow. What you need is a fast dictionary. On Wed, Aug 4, 2010 at 6:00 PM, G?khan Sever wrote: > > > On Wed, Aug 4, 2010 at 6:59 PM, wrote: > >> Hey folks, >> >> I've one array, x, that you could define as follows: >> [[1, 2.25], >> [2, 2.50], >> [3, 2.25], >> [4, 0.00], >> [8, 0.00], >> [9, 2.75]] >> >> Then my second array, y, is: >> [[1, 0.00], >> [2, 0.00], >> [3, 0.00], >> [4, 0.00], >> [5, 0.00], >> [6, 0.00], >> [7, 0.00], >> [8, 0.00], >> [9, 0.00], >> [10,0.00]] >> >> Is there a concise, Numpythonic way to copy the values of x[:,1] over to >> y[:,1] where x[:,0] = y[:,0]? Resulting in, z: >> [[1, 2.25], >> [2, 2.50], >> [3, 2.25], >> [4, 0.00], >> [5, 0.00], >> [6, 0.00], >> [7, 0.00], >> [8, 0.00], >> [9, 2.75], >> [10,0.00]] >> >> My current task has len(x) = 25000 and len(y) = 350000 and looping through >> is quite slow unfortunately. >> >> Many thanks, >> -paul >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > My simplest approach would be: > > y[x[:0]-1] = x > > # Providing the arrays are nicely ordered and 1st column x is all integer. > > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Wed Aug 4 21:22:41 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Wed, 4 Aug 2010 18:22:41 -0700 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: Perhaps try the following: 1) sort x by x[:,0] 2) sort y by y[:,0] 3) loop through both at the same time building an array of indexes A that tells you the index of y[i,0] in x or just building a new array z with the value if you don't need them in order 4) if you do need them in order, unsort A by the sorting used to sort y and then index into x using the unsorted A. use http://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html#numpy.argsort On Wed, Aug 4, 2010 at 6:09 PM, John Salvatier wrote: > How exactly are you looping? That sounds absurdly slow. > > What you need is a fast dictionary. > > On Wed, Aug 4, 2010 at 6:00 PM, G?khan Sever wrote: > >> >> >> On Wed, Aug 4, 2010 at 6:59 PM, wrote: >> >>> Hey folks, >>> >>> I've one array, x, that you could define as follows: >>> [[1, 2.25], >>> [2, 2.50], >>> [3, 2.25], >>> [4, 0.00], >>> [8, 0.00], >>> [9, 2.75]] >>> >>> Then my second array, y, is: >>> [[1, 0.00], >>> [2, 0.00], >>> [3, 0.00], >>> [4, 0.00], >>> [5, 0.00], >>> [6, 0.00], >>> [7, 0.00], >>> [8, 0.00], >>> [9, 0.00], >>> [10,0.00]] >>> >>> Is there a concise, Numpythonic way to copy the values of x[:,1] over to >>> y[:,1] where x[:,0] = y[:,0]? Resulting in, z: >>> [[1, 2.25], >>> [2, 2.50], >>> [3, 2.25], >>> [4, 0.00], >>> [5, 0.00], >>> [6, 0.00], >>> [7, 0.00], >>> [8, 0.00], >>> [9, 2.75], >>> [10,0.00]] >>> >>> My current task has len(x) = 25000 and len(y) = 350000 and looping >>> through is quite slow unfortunately. >>> >>> Many thanks, >>> -paul >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> My simplest approach would be: >> >> y[x[:0]-1] = x >> >> # Providing the arrays are nicely ordered and 1st column x is all integer. >> >> -- >> G?khan >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From PHobson at Geosyntec.com Wed Aug 4 22:24:41 2010 From: PHobson at Geosyntec.com (PHobson at Geosyntec.com) Date: Wed, 4 Aug 2010 22:24:41 -0400 Subject: [Numpy-discussion] Quick array value assignment based on common values In-Reply-To: References: Message-ID: I've deleted the code b/c it was absurdly slow. It was pretty brute-force. -Looped through each row (r) of y -check to see where y[r,0] - x[:,0] < eps (call that row r_hit) -set y[r,1] = x[r_hit,1] There was kind of a short fuse on this, and I was already reading the data from a text file. So I just wrote all of the continuous dates to a file, threw it in a spreadsheet, brought in the other data and did a lookup function that about destroyed my machine. Probably would have been faster to let the looping run [hangs head in shame]. In the future, I'll definitely try the solutions you've outlined. Thanks again! -paul From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of John Salvatier Sent: Wednesday, August 04, 2010 6:23 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Quick array value assignment based on common values Perhaps try the following: 1) sort x by x[:,0] 2) sort y by y[:,0] 3) loop through both at the same time building an array of indexes A that tells you the index of y[i,0] in x or just building a new array z with the value if you don't need them in order 4) if you do need them in order, unsort A by the sorting used to sort y and then index into x using the unsorted A. use http://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html#numpy.argsort On Wed, Aug 4, 2010 at 6:09 PM, John Salvatier > wrote: How exactly are you looping? That sounds absurdly slow. What you need is a fast dictionary. On Wed, Aug 4, 2010 at 6:00 PM, G?khan Sever > wrote: On Wed, Aug 4, 2010 at 6:59 PM, > wrote: Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion My simplest approach would be: y[x[:0]-1] = x # Providing the arrays are nicely ordered and 1st column x is all integer. -- G?khan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej at certik.cz Thu Aug 5 02:06:57 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 4 Aug 2010 23:06:57 -0700 Subject: [Numpy-discussion] update on the transition to git/github In-Reply-To: References: Message-ID: On Tue, Jul 13, 2010 at 7:50 PM, Joshua Holbrook wrote: > This is awesome! I love github. I really wanted to champion for its > use at the BoF but unfortunately missed it. Indeed, that's awesome. I should say finally! :) Ondrej From ralf.gommers at googlemail.com Thu Aug 5 08:45:24 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 Aug 2010 20:45:24 +0800 Subject: [Numpy-discussion] deprecations for 1.5 Message-ID: Hi all, I'm looking at what needs to be deprecated/removed and have a few questions: 1. ma.core has some deprecations without version info: make_mask: flag keyword MaskedArray.flag MaskedArray.raw_data allclose: fillvalue keyword Should these be removed, and if not can someone provide a version number for when that does need to happen? 2. correlate: old_behavior keyword should be removed. Is it correct that that means (besides changes in core/numeric.py) removing multiarray.correlate code in multiarraymodule.c and renaming correlate2 to correlate there? 3. Anything else that needs to be removed/deprecated besides what is noted in the 1.4.0 release notes and ticket 1431? Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Aug 5 10:47:04 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 5 Aug 2010 10:47:04 -0400 Subject: [Numpy-discussion] deprecations for 1.5 In-Reply-To: References: Message-ID: On Aug 5, 2010, at 8:45 AM, Ralf Gommers wrote: > Hi all, > > I'm looking at what needs to be deprecated/removed and have a few questions: > > 1. ma.core has some deprecations without version info: > make_mask: flag keyword > MaskedArray.flag > MaskedArray.raw_data > allclose: fillvalue keyword > Should these be removed, and if not can someone provide a version number for when that does need to happen? Yes, you can remove them, whenever you see fit. From numpy at mspacek.mm.st Thu Aug 5 13:12:50 2010 From: numpy at mspacek.mm.st (Martin Spacek) Date: Thu, 05 Aug 2010 10:12:50 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array Message-ID: <4C5AF112.5090801@mspacek.mm.st> I want to take an n x m array "a" and index into it using an integer index array "i" of length n that will pull out the value at the designated column from each corresponding row of "a". >>> a = np.arange(10) >>> a.shape = 5, 2 >>> a array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]) >>> i = np.array([0, 1, 1, 0, 1]) I want: >>> b = a.foo(i) >>> b array([0, 3, 5, 6, 9]) What's foo? I can't get take() to do what I want. I went and wrote my own little Cython function to do this, but that seems silly (and is also array dtype dependent). I've tried reading through the numpy book, and I'm sure this is somewhere on the list, but I can't find it. I think it has something to do with fancy indexing. I should know how to do this by know... Cheers, Martin From josef.pktd at gmail.com Thu Aug 5 13:26:06 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Aug 2010 13:26:06 -0400 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5AF112.5090801@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> Message-ID: On Thu, Aug 5, 2010 at 1:12 PM, Martin Spacek wrote: > I want to take an n x m array "a" and index into it using an integer index array > "i" of length n that will pull out the value at the designated column from each > corresponding row of "a". > >>>> a = np.arange(10) >>>> a.shape = 5, 2 >>>> a > array([[0, 1], > ? ? ? ?[2, 3], > ? ? ? ?[4, 5], > ? ? ? ?[6, 7], > ? ? ? ?[8, 9]]) >>>> i = np.array([0, 1, 1, 0, 1]) > > I want: > >>>> b = a.foo(i) >>>> b > array([0, 3, 5, 6, 9]) > > What's foo? I can't get take() to do what I want. I went and wrote my own little > Cython function to do this, but that seems silly (and is also array dtype > dependent). I've tried reading through the numpy book, and I'm sure this is > somewhere on the list, but I can't find it. I think it has something to do with > fancy indexing. I should know how to do this by know... like this ? >>> a= np.array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]) >>> i = np.array([0, 1, 1, 0, 1]) >>> a[range(a.shape[0]), i] array([0, 3, 5, 6, 9]) >>> a[np.arange(a.shape[0]), i] array([0, 3, 5, 6, 9]) Josef > > Cheers, > > Martin > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Thu Aug 5 13:26:27 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 5 Aug 2010 10:26:27 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5AF112.5090801@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> Message-ID: On Thu, Aug 5, 2010 at 10:12 AM, Martin Spacek wrote: > I want to take an n x m array "a" and index into it using an integer index array > "i" of length n that will pull out the value at the designated column from each > corresponding row of "a". > >>>> a = np.arange(10) >>>> a.shape = 5, 2 >>>> a > array([[0, 1], > ? ? ? ?[2, 3], > ? ? ? ?[4, 5], > ? ? ? ?[6, 7], > ? ? ? ?[8, 9]]) >>>> i = np.array([0, 1, 1, 0, 1]) > > I want: > >>>> b = a.foo(i) >>>> b > array([0, 3, 5, 6, 9]) Here's one way: >> a.flat[i + a.shape[1] * np.arange(a.shape[0])] array([0, 3, 5, 6, 9]) From kwgoodman at gmail.com Thu Aug 5 13:31:03 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 5 Aug 2010 10:31:03 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> Message-ID: On Thu, Aug 5, 2010 at 10:26 AM, wrote: > On Thu, Aug 5, 2010 at 1:12 PM, Martin Spacek wrote: >> I want to take an n x m array "a" and index into it using an integer index array >> "i" of length n that will pull out the value at the designated column from each >> corresponding row of "a". >> >>>>> a = np.arange(10) >>>>> a.shape = 5, 2 >>>>> a >> array([[0, 1], >> ? ? ? ?[2, 3], >> ? ? ? ?[4, 5], >> ? ? ? ?[6, 7], >> ? ? ? ?[8, 9]]) >>>>> i = np.array([0, 1, 1, 0, 1]) >> >> I want: >> >>>>> b = a.foo(i) >>>>> b >> array([0, 3, 5, 6, 9]) >> >> What's foo? I can't get take() to do what I want. I went and wrote my own little >> Cython function to do this, but that seems silly (and is also array dtype >> dependent). I've tried reading through the numpy book, and I'm sure this is >> somewhere on the list, but I can't find it. I think it has something to do with >> fancy indexing. I should know how to do this by know... > > like this ? Yes, I like it. Textbook case of fancy indexing. > >>>> a= np.array([[0, 1], > ? ? ? [2, 3], > ? ? ? [4, 5], > ? ? ? [6, 7], > ? ? ? [8, 9]]) >>>> i = np.array([0, 1, 1, 0, 1]) >>>> a[range(a.shape[0]), i] > array([0, 3, 5, 6, 9]) >>>> a[np.arange(a.shape[0]), i] > array([0, 3, 5, 6, 9]) > > Josef > >> >> Cheers, >> >> Martin >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jsalvati at u.washington.edu Thu Aug 5 13:47:58 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 5 Aug 2010 10:47:58 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> Message-ID: You may also use the choose function: http://docs.scipy.org/doc/numpy/reference/generated/numpy.choose.html choose(i, (a[:,0], a[:,1]) On Thu, Aug 5, 2010 at 10:31 AM, Keith Goodman wrote: > On Thu, Aug 5, 2010 at 10:26 AM, wrote: > > On Thu, Aug 5, 2010 at 1:12 PM, Martin Spacek > wrote: > >> I want to take an n x m array "a" and index into it using an integer > index array > >> "i" of length n that will pull out the value at the designated column > from each > >> corresponding row of "a". > >> > >>>>> a = np.arange(10) > >>>>> a.shape = 5, 2 > >>>>> a > >> array([[0, 1], > >> [2, 3], > >> [4, 5], > >> [6, 7], > >> [8, 9]]) > >>>>> i = np.array([0, 1, 1, 0, 1]) > >> > >> I want: > >> > >>>>> b = a.foo(i) > >>>>> b > >> array([0, 3, 5, 6, 9]) > >> > >> What's foo? I can't get take() to do what I want. I went and wrote my > own little > >> Cython function to do this, but that seems silly (and is also array > dtype > >> dependent). I've tried reading through the numpy book, and I'm sure this > is > >> somewhere on the list, but I can't find it. I think it has something to > do with > >> fancy indexing. I should know how to do this by know... > > > > like this ? > > Yes, I like it. Textbook case of fancy indexing. > > > > >>>> a= np.array([[0, 1], > > [2, 3], > > [4, 5], > > [6, 7], > > [8, 9]]) > >>>> i = np.array([0, 1, 1, 0, 1]) > >>>> a[range(a.shape[0]), i] > > array([0, 3, 5, 6, 9]) > >>>> a[np.arange(a.shape[0]), i] > > array([0, 3, 5, 6, 9]) > > > > Josef > > > >> > >> Cheers, > >> > >> Martin > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Aug 5 15:06:20 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 5 Aug 2010 15:06:20 -0400 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> <4C59040C.2020901@phys.au.dk> Message-ID: <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> On 2010-08-04, at 2:18 AM, Matthieu Brucher wrote: > 2010/8/4 S?ren Gammelmark : >> >>> >>> I wouldn't know for sure, but could this be related to changes to the >>> gcc compiler in Fedora 13 (with respect to implicit DSO linking) or >>> would that only be an issue at build-time? >>> >>> http://fedoraproject.org/w/index.php?title=UnderstandingDSOLinkChange >> >> I'm not entirely sure I understand the link, but if it has anything to >> do with the compiler it seems to me that it should be the Intel >> compiler. The python I use is compiled with GCC but everything in numpy >> is done with the Intel compilers. Shouldn't it then be something with >> the Intel compilers? >> >> /S?ren > > Unfortunately, I think you'll ahve to use Dag's patch. MKL has a > specific loading procedure since a few releases, you have to abide by > it. I've been having a similar problem compiling NumPy with MKL on a cluster with a site-wide license. Dag's site.cfg fails to config if I use 'iomp5' in it, since (at least with this version, 11.1) libiomp5 is located in /scinet/gpc/intel/Compiler/11.1/072/lib/intel64/ whereas the actual proper MKL /scinet/gpc/intel/Compiler/11.1/072/mkl/lib/em64t/ I've tried putting both in my library_dirs separated by a colon as is suggested by the docs, but python setup.py config fails to find MKL in this case. Has anyone else run into this issue? David From dwf at cs.toronto.edu Thu Aug 5 15:29:07 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 5 Aug 2010 15:29:07 -0400 Subject: [Numpy-discussion] NumPy segfault when running tests (2.7, 1.4.1) In-Reply-To: References: <78CC6B1B-590D-4BF8-BC12-AD4621B0F41D@cs.toronto.edu> Message-ID: <2C52CBC0-0680-4BE1-B22E-C041C3DCE27B@cs.toronto.edu> On 2010-08-03, at 4:09 PM, Pauli Virtanen wrote: > Tue, 03 Aug 2010 15:52:55 -0400, David Warde-Farley wrote: > [clip] >> in PyErr_WarnEx (category=0x11eb6c54, >> text=0x5f90c0 "PyOS_ascii_strtod and PyOS_ascii_atof are deprecated. >> Use PyOS_string_to_double instead.", stack_level=0) at >> Python/_warnings.c:719 >> 719 res = do_warn(message, category, stack_level); >> (gdb) > > That was probably fixed in r8394 in trunk. > > But to be sure, can you supply the whole stack trace (type "bt" in the > gdb prompt). Thanks Pauli, it does seem to be fixed in 1.5.0b1. I didn't get the memo somehow that 2.7 breaks NumPy 1.4.x. Now I just have to chase down ugly MKL problems... David From gokhansever at gmail.com Thu Aug 5 15:43:42 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Thu, 5 Aug 2010 14:43:42 -0500 Subject: [Numpy-discussion] {OT} Mailing trends Message-ID: Hello, There is a nice e-mailing trend tool for Gmail users at http://code.google.com/p/mail-trends/ It is a command line tool producing an html output showing your e-mailing statistics. In my inbox, the following threads are highly ranked in the top threads section. [Numpy-discussion] Announcing toydist, improving distribution and packaging situation [SciPy-Dev] scipy.stats [Numpy-discussion] curious about how people would feel about moving to github Just out of curiosity, are there any mailing trends (top threads, top posters, etc...) provided for the Python related mailing archives? Share your comments please. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy at mspacek.mm.st Thu Aug 5 16:07:50 2010 From: numpy at mspacek.mm.st (Martin Spacek) Date: Thu, 05 Aug 2010 13:07:50 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> Message-ID: <4C5B1A16.6060404@mspacek.mm.st> josef.pkt wrote: >>> a = np.array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]) >>> i = np.array([0, 1, 1, 0, 1]) >>> a[range(a.shape[0]), i] array([0, 3, 5, 6, 9]) >>> a[np.arange(a.shape[0]), i] array([0, 3, 5, 6, 9]) Thanks for all the tips. I guess I was hoping for something that could avoid having to generate np.arange(a.shape[0]), but >>> a[np.arange(a.shape[0]), i] sure is easy to understand. Is there maybe a more CPU and/or memory efficient way? I kind of like John Salvatier's idea: >>> np.choose(i, (a[:,0], a[:,1]) but that would need to be generalized to "a" of arbitrary columns. This could be done using split or vsplit: >>> np.choose(i, np.vsplit(a.T, a.shape[1]))[0] array([0, 3, 5, 6, 9]) That avoids having to generate an np.arange(), but looks kind of wordy. Is there a more compact way? Maybe this is better: >>> b, = i.choose(np.vsplit(a.T, a.shape[1])) >>> b array([0, 3, 5, 6, 9]) Ah, but I've just discovered a strange limitation of choose(): >>> a = np.arange(9*32) >>> a.shape = 9, 32 >>> i = np.random.randint(0, a.shape[1], size=a.shape[0]) >>> i array([ 1, 21, 23, 2, 30, 23, 20, 30, 17]) >>> b, = i.choose(np.vsplit(a.T, a.shape[1])) Traceback (most recent call last): File "", line 1, in ValueError: Need between 2 and (32) array objects (inclusive). Compare with: >>> a = np.arange(9*31) >>> a.shape = 9, 31 >>> i = np.random.randint(0, a.shape[1], size=a.shape[0]) >>> i array([14, 22, 18, 6, 1, 12, 8, 8, 30]) >>> b, = i.choose(np.vsplit(a.T, a.shape[1])) >>> b array([ 14, 53, 80, 99, 125, 167, 194, 225, 278]) So, the ValueError should really read "Need between 2 and 31 array object (inclusive)", should it not? Also, I can't seem to find this limitation in the docs for choose(). I guess I'll stick to using the np.arange(a.shape[0]) method. Martin From josef.pktd at gmail.com Thu Aug 5 16:32:57 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Aug 2010 16:32:57 -0400 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5B1A16.6060404@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> <4C5B1A16.6060404@mspacek.mm.st> Message-ID: On Thu, Aug 5, 2010 at 4:07 PM, Martin Spacek wrote: > josef.pkt wrote: >>>> a = np.array([[0, 1], > ? ? ? ? ? ? ? ? ? [2, 3], > ? ? ? ? ? ? ? ? ? [4, 5], > ? ? ? ? ? ? ? ? ? [6, 7], > ? ? ? ? ? ? ? ? ? [8, 9]]) >>>> i = np.array([0, 1, 1, 0, 1]) >>>> a[range(a.shape[0]), i] > array([0, 3, 5, 6, 9]) >>>> a[np.arange(a.shape[0]), i] > array([0, 3, 5, 6, 9]) > > > Thanks for all the tips. I guess I was hoping for something that could avoid > having to generate np.arange(a.shape[0]), but > > ?>>> a[np.arange(a.shape[0]), i] > > sure is easy to understand. Is there maybe a more CPU and/or memory efficient > way? I kind of like John Salvatier's idea: > > ?>>> np.choose(i, (a[:,0], a[:,1]) > > but that would need to be generalized to "a" of arbitrary columns. seems to work: >>> np.choose(i, a.T) array([0, 3, 5, 6, 9]) but wouldn't get around the 31 limit that you found. Josef This could be > done using split or vsplit: > > ?>>> np.choose(i, np.vsplit(a.T, a.shape[1]))[0] > array([0, 3, 5, 6, 9]) > > That avoids having to generate an np.arange(), but looks kind of wordy. Is there > a more compact way? Maybe this is better: > > ?>>> b, = i.choose(np.vsplit(a.T, a.shape[1])) > ?>>> b > array([0, 3, 5, 6, 9]) > > Ah, but I've just discovered a strange limitation of choose(): > > ?>>> a = np.arange(9*32) > ?>>> a.shape = 9, 32 > ?>>> i = np.random.randint(0, a.shape[1], size=a.shape[0]) > ?>>> i > array([ 1, 21, 23, ?2, 30, 23, 20, 30, 17]) > ?>>> b, = i.choose(np.vsplit(a.T, a.shape[1])) > Traceback (most recent call last): > ? File "", line 1, in > ValueError: Need between 2 and (32) array objects (inclusive). > > Compare with: > > ?>>> a = np.arange(9*31) > ?>>> a.shape = 9, 31 > ?>>> i = np.random.randint(0, a.shape[1], size=a.shape[0]) > ?>>> i > array([14, 22, 18, ?6, ?1, 12, ?8, ?8, 30]) > ?>>> b, = i.choose(np.vsplit(a.T, a.shape[1])) > ?>>> b > array([ 14, ?53, ?80, ?99, 125, 167, 194, 225, 278]) > > So, the ValueError should really read "Need between 2 and 31 array object > (inclusive)", should it not? Also, I can't seem to find this limitation in the > docs for choose(). I guess I'll stick to using the np.arange(a.shape[0]) method. > > Martin > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From bsouthey at gmail.com Thu Aug 5 16:47:46 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 05 Aug 2010 15:47:46 -0500 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5B1A16.6060404@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> <4C5B1A16.6060404@mspacek.mm.st> Message-ID: <4C5B2372.6070402@gmail.com> On 08/05/2010 03:07 PM, Martin Spacek wrote: > josef.pkt wrote: >>>> a = np.array([[0, 1], > [2, 3], > [4, 5], > [6, 7], > [8, 9]]) >>>> i = np.array([0, 1, 1, 0, 1]) >>>> a[range(a.shape[0]), i] > array([0, 3, 5, 6, 9]) >>>> a[np.arange(a.shape[0]), i] > array([0, 3, 5, 6, 9]) > > > Thanks for all the tips. I guess I was hoping for something that could avoid > having to generate np.arange(a.shape[0]), but > > >>> a[np.arange(a.shape[0]), i] > > sure is easy to understand. Is there maybe a more CPU and/or memory efficient > way? I kind of like John Salvatier's idea: > > >>> np.choose(i, (a[:,0], a[:,1]) > > but that would need to be generalized to "a" of arbitrary columns. This could be > done using split or vsplit: > > >>> np.choose(i, np.vsplit(a.T, a.shape[1]))[0] > array([0, 3, 5, 6, 9]) > > That avoids having to generate an np.arange(), but looks kind of wordy. Is there > a more compact way? Maybe this is better: > > >>> b, = i.choose(np.vsplit(a.T, a.shape[1])) > >>> b > array([0, 3, 5, 6, 9]) > > Ah, but I've just discovered a strange limitation of choose(): > > >>> a = np.arange(9*32) > >>> a.shape = 9, 32 > >>> i = np.random.randint(0, a.shape[1], size=a.shape[0]) > >>> i > array([ 1, 21, 23, 2, 30, 23, 20, 30, 17]) > >>> b, = i.choose(np.vsplit(a.T, a.shape[1])) > Traceback (most recent call last): > File "", line 1, in > ValueError: Need between 2 and (32) array objects (inclusive). > > Compare with: > > >>> a = np.arange(9*31) > >>> a.shape = 9, 31 > >>> i = np.random.randint(0, a.shape[1], size=a.shape[0]) > >>> i > array([14, 22, 18, 6, 1, 12, 8, 8, 30]) > >>> b, = i.choose(np.vsplit(a.T, a.shape[1])) > >>> b > array([ 14, 53, 80, 99, 125, 167, 194, 225, 278]) > > So, the ValueError should really read "Need between 2 and 31 array object > (inclusive)", should it not? Also, I can't seem to find this limitation in the > docs for choose(). I guess I'll stick to using the np.arange(a.shape[0]) method. > > Martin > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion I think you might want numpy's where function: >>> np.where(i,a[:,1],a[:,0]) array([0, 3, 5, 6, 9]) Bruce From kwgoodman at gmail.com Thu Aug 5 16:51:39 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 5 Aug 2010 13:51:39 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> <4C5B1A16.6060404@mspacek.mm.st> Message-ID: On Thu, Aug 5, 2010 at 1:32 PM, wrote: > On Thu, Aug 5, 2010 at 4:07 PM, Martin Spacek wrote: >> josef.pkt wrote: >>>>> a = np.array([[0, 1], >> ? ? ? ? ? ? ? ? ? [2, 3], >> ? ? ? ? ? ? ? ? ? [4, 5], >> ? ? ? ? ? ? ? ? ? [6, 7], >> ? ? ? ? ? ? ? ? ? [8, 9]]) >>>>> i = np.array([0, 1, 1, 0, 1]) >>>>> a[range(a.shape[0]), i] >> array([0, 3, 5, 6, 9]) >>>>> a[np.arange(a.shape[0]), i] >> array([0, 3, 5, 6, 9]) >> >> >> Thanks for all the tips. I guess I was hoping for something that could avoid >> having to generate np.arange(a.shape[0]), but >> >> ?>>> a[np.arange(a.shape[0]), i] >> >> sure is easy to understand. Is there maybe a more CPU and/or memory efficient >> way? I kind of like John Salvatier's idea: >> >> ?>>> np.choose(i, (a[:,0], a[:,1]) >> >> but that would need to be generalized to "a" of arbitrary columns. > > seems to work: > >>>> np.choose(i, a.T) > array([0, 3, 5, 6, 9]) > > but wouldn't get around the 31 limit that you found. Choose is fast: >> N = 1000 >> a = np.random.randint(0, 9, (N,2)) >> i = np.random.randint(0, 2, N) >> timeit a[range(N), i] 10000 loops, best of 3: 108 us per loop >> timeit a[np.arange(N), i] 10000 loops, best of 3: 39 us per loop >> timeit np.choose(i, a.T) 10000 loops, best of 3: 32.3 us per loop But flat is faster: >> timeit idx = np.arange(N); idx *= 2; idx += i; a.flat[idx] 100000 loops, best of 3: 16.8 us per loop From gammelmark at phys.au.dk Thu Aug 5 16:53:58 2010 From: gammelmark at phys.au.dk (=?ISO-8859-1?Q?S=F8ren_Gammelmark?=) Date: Thu, 05 Aug 2010 22:53:58 +0200 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> <4C59040C.2020901@phys.au.dk> <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> Message-ID: <4C5B24E6.7080508@phys.au.dk> It seems to me, that you are using an libiomp5 for Intel Itanium (lib/intel64) or such, but an MKL for EM64T-processors (lib/em64t). In my case I used EM64T in all cases (I'm running AMD Opteron) . I don't think the two types of libraries are compatible, but I might be wrong. /S?ren On 05-08-2010 21:06, David Warde-Farley wrote: > On 2010-08-04, at 2:18 AM, Matthieu Brucher wrote: > > >> 2010/8/4 S?ren Gammelmark: >> >>> >>>> I wouldn't know for sure, but could this be related to changes to the >>>> gcc compiler in Fedora 13 (with respect to implicit DSO linking) or >>>> would that only be an issue at build-time? >>>> >>>> http://fedoraproject.org/w/index.php?title=UnderstandingDSOLinkChange >>>> >>> I'm not entirely sure I understand the link, but if it has anything to >>> do with the compiler it seems to me that it should be the Intel >>> compiler. The python I use is compiled with GCC but everything in numpy >>> is done with the Intel compilers. Shouldn't it then be something with >>> the Intel compilers? >>> >>> /S?ren >>> >> Unfortunately, I think you'll ahve to use Dag's patch. MKL has a >> specific loading procedure since a few releases, you have to abide by >> it. >> > I've been having a similar problem compiling NumPy with MKL on a cluster with a site-wide license. Dag's site.cfg fails to config if I use 'iomp5' in it, since (at least with this version, 11.1) libiomp5 is located in > > /scinet/gpc/intel/Compiler/11.1/072/lib/intel64/ > > whereas the actual proper MKL > > /scinet/gpc/intel/Compiler/11.1/072/mkl/lib/em64t/ > > I've tried putting both in my library_dirs separated by a colon as is suggested by the docs, but python setup.py config fails to find MKL in this case. Has anyone else run into this issue? > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Thu Aug 5 16:55:04 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Aug 2010 16:55:04 -0400 Subject: [Numpy-discussion] {OT} Mailing trends In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 3:43 PM, G?khan Sever wrote: > Hello, > There is a nice e-mailing trend tool for Gmail users > at?http://code.google.com/p/mail-trends/ > It is a command line tool producing an html output showing your e-mailing > statistics. In my inbox, the following threads are highly ranked in the top > threads section. > > [Numpy-discussion] Announcing toydist, improving distribution and packaging > situation > [SciPy-Dev] scipy.stats > [Numpy-discussion] curious about how people would feel about moving to > github > > Just out of curiosity, are there any mailing trends (top threads, top > posters, etc...) provided for the Python related mailing archives? > Share your comments please. I only know the top poster statistics for googlegroups http://groups.google.ca/group/scipy-user/about?hl=en but numpy-discussion and scipy-dev are not on google groups Josef > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From jsalvati at u.washington.edu Thu Aug 5 17:00:19 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 5 Aug 2010 14:00:19 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> <4C5B1A16.6060404@mspacek.mm.st> Message-ID: choose might be slower if you weren't doing an "arange(N)" each time. On Thu, Aug 5, 2010 at 1:51 PM, Keith Goodman wrote: > On Thu, Aug 5, 2010 at 1:32 PM, wrote: > > On Thu, Aug 5, 2010 at 4:07 PM, Martin Spacek > wrote: > >> josef.pkt wrote: > >>>>> a = np.array([[0, 1], > >> [2, 3], > >> [4, 5], > >> [6, 7], > >> [8, 9]]) > >>>>> i = np.array([0, 1, 1, 0, 1]) > >>>>> a[range(a.shape[0]), i] > >> array([0, 3, 5, 6, 9]) > >>>>> a[np.arange(a.shape[0]), i] > >> array([0, 3, 5, 6, 9]) > >> > >> > >> Thanks for all the tips. I guess I was hoping for something that could > avoid > >> having to generate np.arange(a.shape[0]), but > >> > >> >>> a[np.arange(a.shape[0]), i] > >> > >> sure is easy to understand. Is there maybe a more CPU and/or memory > efficient > >> way? I kind of like John Salvatier's idea: > >> > >> >>> np.choose(i, (a[:,0], a[:,1]) > >> > >> but that would need to be generalized to "a" of arbitrary columns. > > > > seems to work: > > > >>>> np.choose(i, a.T) > > array([0, 3, 5, 6, 9]) > > > > but wouldn't get around the 31 limit that you found. > > Choose is fast: > > >> N = 1000 > >> a = np.random.randint(0, 9, (N,2)) > >> i = np.random.randint(0, 2, N) > >> timeit a[range(N), i] > 10000 loops, best of 3: 108 us per loop > >> timeit a[np.arange(N), i] > 10000 loops, best of 3: 39 us per loop > >> timeit np.choose(i, a.T) > 10000 loops, best of 3: 32.3 us per loop > > But flat is faster: > > >> timeit idx = np.arange(N); idx *= 2; idx += i; a.flat[idx] > 100000 loops, best of 3: 16.8 us per loop > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Aug 5 17:18:44 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 5 Aug 2010 23:18:44 +0200 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> <4C59040C.2020901@phys.au.dk> <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> Message-ID: > I've been having a similar problem compiling NumPy with MKL on a cluster with a site-wide license. Dag's site.cfg fails to config if I use 'iomp5' in it, since (at least with this version, 11.1) libiomp5 is located in > > ? ? ? ?/scinet/gpc/intel/Compiler/11.1/072/lib/intel64/ > > whereas the actual proper MKL > > ? ? ? ?/scinet/gpc/intel/Compiler/11.1/072/mkl/lib/em64t/ > > I've tried putting both in my library_dirs separated by a colon as is suggested by the docs, but python setup.py config fails to find MKL in this case. Has anyone else run into this issue? Indeed, this is a issue I also faced. I had to copy all the libs in one folder. If one has only MKL, iomp5 is provided (as well as guide IIRC), but with the Compiler pack, they are not in the MKL lib folder. Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From dwf at cs.toronto.edu Thu Aug 5 18:05:50 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 5 Aug 2010 18:05:50 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: <4FE4E2B9-7196-4356-8213-8358C783BF78@cs.toronto.edu> On 2010-08-01, at 12:38 PM, Ralf Gommers wrote: > I am pleased to announce the availability of the first beta of NumPy 1.5.0. This will be the first NumPy release to include support for Python 3, as well as for Python 2.7. Please try this beta and report any problems on the NumPy mailing list. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > Please note that binaries for Python 3.1 are not yet up, they will follow as soon as a minor issue with building them is resolved. Building from source with Python 3.1 should work without problems. Hey Ralf, I am getting a single test failure: ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/dwf/pkg/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 275, in test_special_values assert_almost_equal(np.log(np.conj(xa[i])), np.conj(np.log(xa[i]))) File "/home/dwf/pkg/lib/python2.7/site-packages/numpy/testing/utils.py", line 443, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: array([-inf+3.14159265j]) DESIRED: array([-inf-3.14159265j]) >> raise AssertionError('\nArrays are not almost equal\n ACTUAL: array([-inf+3.14159265j])\n DESIRED: array([-inf-3.14159265j])') This is on a Xeon E5540 built against the MKL 11.1, if that matters (I suspect it doesn't). David -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Aug 5 18:17:20 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 5 Aug 2010 18:17:20 -0400 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <4C5B24E6.7080508@phys.au.dk> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> <4C59040C.2020901@phys.au.dk> <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> <4C5B24E6.7080508@phys.au.dk> Message-ID: <3EB52115-BF5E-49F4-AAAE-2E9DC3EEDB0B@cs.toronto.edu> On 2010-08-05, at 4:53 PM, S?ren Gammelmark wrote: > It seems to me, that you are using an libiomp5 for Intel Itanium > (lib/intel64) or such, but an MKL for EM64T-processors (lib/em64t). In > my case I used EM64T in all cases (I'm running AMD Opteron) . I don't > think the two types of libraries are compatible, but I might be wrong. I don't think lib/intel64 implies Itanium. Indeed, running 'file' on it yields libiomp5.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped I found this very helpful blog post: http://marklodato.github.com/2009/08/30/numpy-scipy-and-intel.html which seems to get NumPy built for me on the cluster. The key seemed to be changing the 'icc' executable and also explicitly selecting the mkl_mc (or in my case mc3) library to link. David From numpy at mspacek.mm.st Fri Aug 6 06:01:10 2010 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 06 Aug 2010 03:01:10 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> Message-ID: <4C5BDD66.9030002@mspacek.mm.st> Keith Goodman wrote: > Here's one way: > >>> a.flat[i + a.shape[1] * np.arange(a.shape[0])] > array([0, 3, 5, 6, 9]) I'm afraid I made my example a little too simple. In retrospect, what I really want is to be able to use a 2D index array "i", like this: >>> a = np.array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19]]) >>> i = np.array([[2, 1], [3, 1], [1, 1], [0, 0], [3, 1]]) >>> foo(a, i) array([[ 2, 1], [ 7, 5], [ 9, 9], [12, 12], [19, 17]]) I think the flat iterator indexing suggestion is about the only thing that'll work. Here's the function I've pretty much settled on: def rowtake(a, i): """For each row in a, return values according to column indices in the corresponding row in i. Returned shape == i.shape""" assert a.ndim == 2 assert i.ndim <= 2 if i.ndim == 1: return a.flat[i + a.shape[1] * np.arange(a.shape[0])] else: # i.ndim == 2 return a.flat[i + a.shape[1] * np.vstack(np.arange(a.shape[0]))] This is about half as fast as my Cython function, but the Cython function is limited to fixed dtypes and ndim: @cython.boundscheck(False) @cython.wraparound(False) @cython.cdivision(True) def rowtake_cy(np.ndarray[np.int32_t, ndim=2] a, np.ndarray[np.int32_t, ndim=2] i): """For each row in a, return values according to column indices in the corresponding row in i. Returned shape == i.shape""" cdef Py_ssize_t nrows, ncols, rowi, coli cdef np.ndarray[np.int32_t, ndim=2] out nrows = i.shape[0] ncols = i.shape[1] # num cols to take from a for each row assert a.shape[0] == nrows assert i.max() < a.shape[1] out = np.empty((nrows, ncols), dtype=np.int32) for rowi in range(nrows): for coli in range(ncols): out[rowi, coli] = a[rowi, i[rowi, coli]] return out Cheers, Martin From josef.pktd at gmail.com Fri Aug 6 06:29:25 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Aug 2010 06:29:25 -0400 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5BDD66.9030002@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> <4C5BDD66.9030002@mspacek.mm.st> Message-ID: On Fri, Aug 6, 2010 at 6:01 AM, Martin Spacek wrote: > Keith Goodman wrote: > ?> Here's one way: > ?> > ?>>> a.flat[i + a.shape[1] * np.arange(a.shape[0])] > ?> ? ? array([0, 3, 5, 6, 9]) > > > I'm afraid I made my example a little too simple. In retrospect, what I really > want is to be able to use a 2D index array "i", like this: > > ?>>> a = np.array([[ 0, ?1, ?2, ?3], > ? ? ? ? ? ? ? ? ? [ 4, ?5, ?6, ?7], > ? ? ? ? ? ? ? ? ? [ 8, ?9, 10, 11], > ? ? ? ? ? ? ? ? ? [12, 13, 14, 15], > ? ? ? ? ? ? ? ? ? [16, 17, 18, 19]]) > ?>>> i = np.array([[2, 1], > ? ? ? ? ? ? ? ? ? [3, 1], > ? ? ? ? ? ? ? ? ? [1, 1], > ? ? ? ? ? ? ? ? ? [0, 0], > ? ? ? ? ? ? ? ? ? [3, 1]]) > ?>>> foo(a, i) > array([[ 2, ?1], > ? ? ? ?[ 7, ?5], > ? ? ? ?[ 9, ?9], > ? ? ? ?[12, 12], > ? ? ? ?[19, 17]]) > > I think the flat iterator indexing suggestion is about the only thing that'll > work. Here's the function I've pretty much settled on: > > def rowtake(a, i): > ? ? """For each row in a, return values according to column indices in the > ? ? corresponding row in i. Returned shape == i.shape""" > ? ? assert a.ndim == 2 > ? ? assert i.ndim <= 2 > ? ? if i.ndim == 1: > ? ? ? ? return a.flat[i + a.shape[1] * np.arange(a.shape[0])] > ? ? else: # i.ndim == 2 > ? ? ? ? return a.flat[i + a.shape[1] * np.vstack(np.arange(a.shape[0]))] I still find broadcasting easier to read, even if it might be a bit slower >>> a[np.arange(5)[:,None], i] array([[ 2, 1], [ 7, 5], [ 9, 9], [12, 12], [19, 17]]) Josef > > This is about half as fast as my Cython function, but the Cython function is > limited to fixed dtypes and ndim: > > @cython.boundscheck(False) > @cython.wraparound(False) > @cython.cdivision(True) > def rowtake_cy(np.ndarray[np.int32_t, ndim=2] a, > ? ? ? ? ? ? ? ?np.ndarray[np.int32_t, ndim=2] i): > ? ? """For each row in a, return values according to column indices in the > ? ? corresponding row in i. Returned shape == i.shape""" > > ? ? cdef Py_ssize_t nrows, ncols, rowi, coli > ? ? cdef np.ndarray[np.int32_t, ndim=2] out > > ? ? nrows = i.shape[0] > ? ? ncols = i.shape[1] # num cols to take from a for each row > ? ? assert a.shape[0] == nrows > ? ? assert i.max() < a.shape[1] > ? ? out = np.empty((nrows, ncols), dtype=np.int32) > > ? ? for rowi in range(nrows): > ? ? ? ? for coli in range(ncols): > ? ? ? ? ? ? out[rowi, coli] = a[rowi, i[rowi, coli]] > > ? ? return out > > Cheers, > > Martin > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Fri Aug 6 09:57:40 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 6 Aug 2010 06:57:40 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5BDD66.9030002@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> <4C5BDD66.9030002@mspacek.mm.st> Message-ID: On Fri, Aug 6, 2010 at 3:01 AM, Martin Spacek wrote: > Keith Goodman wrote: > ?> Here's one way: > ?> > ?>>> a.flat[i + a.shape[1] * np.arange(a.shape[0])] > ?> ? ? array([0, 3, 5, 6, 9]) > > > I'm afraid I made my example a little too simple. In retrospect, what I really > want is to be able to use a 2D index array "i", like this: > > ?>>> a = np.array([[ 0, ?1, ?2, ?3], > ? ? ? ? ? ? ? ? ? [ 4, ?5, ?6, ?7], > ? ? ? ? ? ? ? ? ? [ 8, ?9, 10, 11], > ? ? ? ? ? ? ? ? ? [12, 13, 14, 15], > ? ? ? ? ? ? ? ? ? [16, 17, 18, 19]]) > ?>>> i = np.array([[2, 1], > ? ? ? ? ? ? ? ? ? [3, 1], > ? ? ? ? ? ? ? ? ? [1, 1], > ? ? ? ? ? ? ? ? ? [0, 0], > ? ? ? ? ? ? ? ? ? [3, 1]]) > ?>>> foo(a, i) > array([[ 2, ?1], > ? ? ? ?[ 7, ?5], > ? ? ? ?[ 9, ?9], > ? ? ? ?[12, 12], > ? ? ? ?[19, 17]]) > > I think the flat iterator indexing suggestion is about the only thing that'll > work. Here's the function I've pretty much settled on: > > def rowtake(a, i): > ? ? """For each row in a, return values according to column indices in the > ? ? corresponding row in i. Returned shape == i.shape""" > ? ? assert a.ndim == 2 > ? ? assert i.ndim <= 2 > ? ? if i.ndim == 1: > ? ? ? ? return a.flat[i + a.shape[1] * np.arange(a.shape[0])] > ? ? else: # i.ndim == 2 > ? ? ? ? return a.flat[i + a.shape[1] * np.vstack(np.arange(a.shape[0]))] > > This is about half as fast as my Cython function, but the Cython function is > limited to fixed dtypes and ndim: You can speed it up by getting rid of two copies: idx = np.arange(a.shape[0]) idx *= a.shape[1] idx += i From vincent at vincentdavis.net Fri Aug 6 11:13:33 2010 From: vincent at vincentdavis.net (Vincent Davis) Date: Fri, 6 Aug 2010 09:13:33 -0600 Subject: [Numpy-discussion] {OT} Mailing trends In-Reply-To: References: Message-ID: On Thu, Aug 5, 2010 at 2:55 PM, wrote: > On Thu, Aug 5, 2010 at 3:43 PM, G?khan Sever > wrote: > > Hello, > > There is a nice e-mailing trend tool for Gmail users > > at http://code.google.com/p/mail-trends/ > > It is a command line tool producing an html output showing your e-mailing > > statistics. In my inbox, the following threads are highly ranked in the > top > > threads section. > > > > [Numpy-discussion] Announcing toydist, improving distribution and > packaging > > situation > > [SciPy-Dev] scipy.stats > > [Numpy-discussion] curious about how people would feel about moving to > > github > > > > Just out of curiosity, are there any mailing trends (top threads, top > > posters, etc...) provided for the Python related mailing archives? > > Share your comments please. > > I only know the top poster statistics for googlegroups > > http://groups.google.ca/group/scipy-user/about?hl=en > > but numpy-discussion and scipy-dev are not on google groups > > Is scipy-user a google group or just mirrored? As a side note or idea I was playing with trying to search for each function/module in the archive as a way to rank/prioritize what documentation may need improvement. For example searching http://groups.google.ca/group/scipy-user/about?hl=en *38* results for * randint** *6,950 results for optimize** Of course there are many reasons an it just might be popular in example code and not really the question in the post. Vincent Josef > > > > -- > > G?khan > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > *Vincent Davis 720-301-3003 * vincent at vincentdavis.net my blog | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.becker at amolf.nl Fri Aug 6 11:46:53 2010 From: n.becker at amolf.nl (Nils Becker) Date: Fri, 06 Aug 2010 17:46:53 +0200 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) Message-ID: <4C5C2E6D.30403@amolf.nl> Hi, I found what looks like a bug in histogram, when the option normed=True is used together with non-uniform bins. Consider this example: import numpy as np data = np.array([1, 2, 3, 4]) bins = np.array([.5, 1.5, 4.5]) bin_widths = np.diff(bins) (counts, dummy) = np.histogram(data, bins) (densities, dummy) = np.histogram(data, bins, normed=True) What this gives is: bin_widths array([ 1., 3.]) counts array([1, 3]) densities array([ 0.1, 0.3]) The documentation claims that histogram with normed=True gives a density, which integrates to 1. In this example, it is true that (densities * bin_widths).sum() is 1. However, clearly the data are equally spaced, so their density should be uniform and equal to 0.25. Note that (0.25 * bin_widths).sum() is also 1. I believe np.histogram(data, bins, normed=True) effectively does : np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). However, it _should_ do np.histogram(data, bins, normed=False) / bins_widths to get a true density over the data coordinate as a result. It's easy to fix by hand, but I think the documentation is at least misleading?! sorry if this has been discussed before; I did not find it anyway (numpy 1.3) From josef.pktd at gmail.com Fri Aug 6 12:16:07 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Aug 2010 12:16:07 -0400 Subject: [Numpy-discussion] {OT} Mailing trends In-Reply-To: References: Message-ID: On Fri, Aug 6, 2010 at 11:13 AM, Vincent Davis wrote: > > On Thu, Aug 5, 2010 at 2:55 PM, wrote: >> >> On Thu, Aug 5, 2010 at 3:43 PM, G?khan Sever wrote: >> > Hello, >> > There is a nice e-mailing trend tool for Gmail users >> > at?http://code.google.com/p/mail-trends/ >> > It is a command line tool producing an html output showing your e-mailing >> > statistics. In my inbox, the following threads are highly ranked in the top >> > threads section. >> > >> > [Numpy-discussion] Announcing toydist, improving distribution and packaging >> > situation >> > [SciPy-Dev] scipy.stats >> > [Numpy-discussion] curious about how people would feel about moving to >> > github >> > >> > Just out of curiosity, are there any mailing trends (top threads, top >> > posters, etc...) provided for the Python related mailing archives? >> > Share your comments please. >> >> I only know the top poster statistics for googlegroups >> >> http://groups.google.ca/group/scipy-user/about?hl=en >> >> but numpy-discussion and scipy-dev are not on google groups >> > > Is scipy-user a google group or just mirrored? just mirrored, original is on scipy.org > > As a side note or idea I was playing with trying to search for each function/module in the archive as a way to rank/prioritize what documentation may need improvement. > For example searching?http://groups.google.ca/group/scipy-user/about?hl=en > 38 results for randint* > 6,950 results for optimize* > Of course there are many reasons an it just might be popular in example code and not really the question in the post. I didn't know adding stars works: 12,600 results for *stats* 120 results for *stats* bug I would guess that the most popular functions are also the ones that are the best documented (or they are popular because they have the least obvious API). (for example I didn't find much on ttest, too obvious ? except for a possible enhancement/addition http://groups.google.ca/group/scipy-user/browse_thread/thread/bc3c36f8908a20af/a85a5d6b7d457436?hl=en&lnk=gst&q=t_test*#a85a5d6b7d457436 ) Josef > > Vincent >> >> Josef >> >> >> > -- >> > G?khan >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Vincent Davis > 720-301-3003 > vincent at vincentdavis.net > > my blog | LinkedIn > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Fri Aug 6 12:37:45 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Aug 2010 12:37:45 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: <4C5C2E6D.30403@amolf.nl> References: <4C5C2E6D.30403@amolf.nl> Message-ID: On Fri, Aug 6, 2010 at 11:46 AM, Nils Becker wrote: > Hi, > > I found what looks like a bug in histogram, when the option normed=True > is used together with non-uniform bins. > > Consider this example: > > import numpy as np > data = np.array([1, 2, 3, 4]) > bins = np.array([.5, 1.5, 4.5]) > bin_widths = np.diff(bins) > (counts, dummy) = np.histogram(data, bins) > (densities, dummy) = np.histogram(data, bins, normed=True) > > What this gives is: > > bin_widths > array([ 1., ?3.]) > > counts > array([1, 3]) > > densities > array([ 0.1, ?0.3]) > > The documentation claims that histogram with normed=True gives a > density, which integrates to 1. In this example, it is true that > (densities * bin_widths).sum() is 1. However, clearly the data are > equally spaced, so their density should be uniform and equal to 0.25. > Note that (0.25 * bin_widths).sum() is also 1. > > I believe np.histogram(data, bins, normed=True) effectively does : > np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). > > However, it _should_ do > np.histogram(data, bins, normed=False) / bins_widths > > to get a true density over the data coordinate as a result. It's easy to > fix by hand, but I think the documentation is at least misleading?! > > sorry if this has been discussed before; I did not find it anyway (numpy > 1.3) Either I also don't understand histogram or this is a bug. >>> data = np.arange(1,10) >>> bins = np.array([.5, 1.5, 4.5, 7.5, 8.5, 9.5]) >>> np.histogram(data, bins, normed=True) (array([ 0.04761905, 0.14285714, 0.14285714, 0.04761905, 0.04761905]), array([ 0.5, 1.5, 4.5, 7.5, 8.5, 9.5])) >>> np.histogram(data, bins) (array([1, 3, 3, 1, 1]), array([ 0.5, 1.5, 4.5, 7.5, 8.5, 9.5])) >>> np.diff(bins) array([ 1., 3., 3., 1., 1.]) I don't see what the normed=True numbers are in this case. >>> np.array([ 1., 3., 3., 1., 1.])/7 array([ 0.14285714, 0.42857143, 0.42857143, 0.14285714, 0.14285714]) Josef > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From faltet at pytables.org Fri Aug 6 14:14:11 2010 From: faltet at pytables.org (Francesc Alted) Date: Fri, 6 Aug 2010 19:14:11 +0100 Subject: [Numpy-discussion] Installing numpy with MKL In-Reply-To: <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> References: <4C582DD0.4050505@phys.au.dk> <4C5844F0.10907@student.matnat.uio.no> <4C587CAD.4040204@phys.au.dk> <4C59040C.2020901@phys.au.dk> <118BDF1F-74EE-411E-99BB-B45DE8F08739@cs.toronto.edu> Message-ID: 2010/8/5, David Warde-Farley : > I've been having a similar problem compiling NumPy with MKL on a cluster > with a site-wide license. Dag's site.cfg fails to config if I use 'iomp5' in > it, since (at least with this version, 11.1) libiomp5 is located in > > /scinet/gpc/intel/Compiler/11.1/072/lib/intel64/ > > whereas the actual proper MKL > > /scinet/gpc/intel/Compiler/11.1/072/mkl/lib/em64t/ > > I've tried putting both in my library_dirs separated by a colon as is > suggested by the docs, but python setup.py config fails to find MKL in this > case. Has anyone else run into this issue? I've made a patch to solve this some time ago: http://projects.scipy.org/numpy/ticket/993 but it did not make into the repo yet. -- Francesc Alted From bsouthey at gmail.com Fri Aug 6 14:22:40 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 06 Aug 2010 13:22:40 -0500 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C2E6D.30403@amolf.nl> Message-ID: <4C5C52F0.7080409@gmail.com> On 08/06/2010 11:37 AM, josef.pktd at gmail.com wrote: > On Fri, Aug 6, 2010 at 11:46 AM, Nils Becker wrote: >> Hi, >> >> I found what looks like a bug in histogram, when the option normed=True >> is used together with non-uniform bins. >> >> Consider this example: >> >> import numpy as np >> data = np.array([1, 2, 3, 4]) >> bins = np.array([.5, 1.5, 4.5]) >> bin_widths = np.diff(bins) >> (counts, dummy) = np.histogram(data, bins) >> (densities, dummy) = np.histogram(data, bins, normed=True) >> >> What this gives is: >> >> bin_widths >> array([ 1., 3.]) >> >> counts >> array([1, 3]) >> >> densities >> array([ 0.1, 0.3]) >> >> The documentation claims that histogram with normed=True gives a >> density, which integrates to 1. In this example, it is true that >> (densities * bin_widths).sum() is 1. However, clearly the data are >> equally spaced, so their density should be uniform and equal to 0.25. >> Note that (0.25 * bin_widths).sum() is also 1. >> >> I believe np.histogram(data, bins, normed=True) effectively does : >> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). >> >> However, it _should_ do >> np.histogram(data, bins, normed=False) / bins_widths >> >> to get a true density over the data coordinate as a result. It's easy to >> fix by hand, but I think the documentation is at least misleading?! >> >> sorry if this has been discussed before; I did not find it anyway (numpy >> 1.3) > Either I also don't understand histogram or this is a bug. > >>>> data = np.arange(1,10) >>>> bins = np.array([.5, 1.5, 4.5, 7.5, 8.5, 9.5]) >>>> np.histogram(data, bins, normed=True) > (array([ 0.04761905, 0.14285714, 0.14285714, 0.04761905, > 0.04761905]), array([ 0.5, 1.5, 4.5, 7.5, 8.5, 9.5])) >>>> np.histogram(data, bins) > (array([1, 3, 3, 1, 1]), array([ 0.5, 1.5, 4.5, 7.5, 8.5, 9.5])) >>>> np.diff(bins) > array([ 1., 3., 3., 1., 1.]) > > I don't see what the normed=True numbers are in this case. > >>>> np.array([ 1., 3., 3., 1., 1.])/7 > array([ 0.14285714, 0.42857143, 0.42857143, 0.14285714, 0.14285714]) > > Josef > > As I recall, there as issues with this aspect. Please search the discussion regarding histogram especially David Huard's reply in this thread: http://thread.gmane.org/gmane.comp.python.numeric.general/22445 Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Fri Aug 6 15:30:33 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 6 Aug 2010 14:30:33 -0500 Subject: [Numpy-discussion] Broken links on new.scipy Message-ID: Hi, @ http://new.scipy.org/download.html numpy and scipy links for Fedora is broken. Could you update the links with these? https://admin.fedoraproject.org/pkgdb/acls/name/numpy https://admin.fedoraproject.org/pkgdb/acls/name/scipy Thanks. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vincent at vincentdavis.net Fri Aug 6 16:00:04 2010 From: vincent at vincentdavis.net (Vincent Davis) Date: Fri, 6 Aug 2010 14:00:04 -0600 Subject: [Numpy-discussion] {OT} Mailing trends In-Reply-To: References: Message-ID: On Fri, Aug 6, 2010 at 10:16 AM, wrote: > On Fri, Aug 6, 2010 at 11:13 AM, Vincent Davis > wrote: > > > > On Thu, Aug 5, 2010 at 2:55 PM, wrote: > >> > >> On Thu, Aug 5, 2010 at 3:43 PM, G?khan Sever > wrote: > >> > Hello, > >> > There is a nice e-mailing trend tool for Gmail users > >> > at http://code.google.com/p/mail-trends/ > >> > It is a command line tool producing an html output showing your > e-mailing > >> > statistics. In my inbox, the following threads are highly ranked in > the top > >> > threads section. > >> > > >> > [Numpy-discussion] Announcing toydist, improving distribution and > packaging > >> > situation > >> > [SciPy-Dev] scipy.stats > >> > [Numpy-discussion] curious about how people would feel about moving to > >> > github > >> > > >> > Just out of curiosity, are there any mailing trends (top threads, top > >> > posters, etc...) provided for the Python related mailing archives? > >> > Share your comments please. > >> > >> I only know the top poster statistics for googlegroups > >> > >> http://groups.google.ca/group/scipy-user/about?hl=en > >> > >> but numpy-discussion and scipy-dev are not on google groups > >> > > > > Is scipy-user a google group or just mirrored? > > just mirrored, original is on scipy.org > > > > > As a side note or idea I was playing with trying to search for each > function/module in the archive as a way to rank/prioritize what > documentation may need improvement. > > For example searching > http://groups.google.ca/group/scipy-user/about?hl=en > > 38 results for randint* > > 6,950 results for optimize* > > Of course there are many reasons an it just might be popular in example > code and not really the question in the post. > > I didn't know adding stars works: > 12,600 results for *stats* > 120 results for *stats* bug > > I would guess that the most popular functions are also the ones that > are the best documented (or they are popular because they have the > least obvious API). > > This is cool. http://groups.google.ca/advanced_search?q= So the google group is just a mirror of the mailman list? I was thinking this might be valid way to prioritize the documentation (if one wanted to do such a thing) 1; the documentation is poor, or there is no example (based on a quick look) 2; prioritize based on hits in the mail list. Vincent (for example I didn't find much on ttest, too obvious ? except for a > possible enhancement/addition > > http://groups.google.ca/group/scipy-user/browse_thread/thread/bc3c36f8908a20af/a85a5d6b7d457436?hl=en&lnk=gst&q=t_test*#a85a5d6b7d457436 > ) > > Josef > > > > > Vincent > >> > >> Josef > >> > >> > >> > -- > >> > G?khan > >> > > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > Vincent Davis > > 720-301-3003 > > vincent at vincentdavis.net > > > > my blog | LinkedIn > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > *Vincent Davis 720-301-3003 * vincent at vincentdavis.net my blog | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy at mspacek.mm.st Fri Aug 6 16:11:55 2010 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 06 Aug 2010 13:11:55 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: References: <4C5AF112.5090801@mspacek.mm.st> <4C5BDD66.9030002@mspacek.mm.st> Message-ID: <4C5C6C8B.9030006@mspacek.mm.st> On 2010-08-06 06:57, Keith Goodman wrote: > You can speed it up by getting rid of two copies: > > idx = np.arange(a.shape[0]) > idx *= a.shape[1] > idx += i Keith, you're right of course. I'd forgotten about your earlier suggestion about operating in-place. Here's my new version: def rowtake(a, i): """For each row in a, return values according to column indices in the corresponding row in i. Returned shape == i.shape""" assert a.ndim == 2 assert i.ndim <= 2 if i.ndim == 1: j = np.arange(a.shape[0]) else: # i.ndim == 2 j = np.repeat(np.arange(a.shape[0]), i.shape[1]) j.shape = i.shape j *= a.shape[1] j += i return a.flat[j] >>> a = np.arange(20) >>> a.shape = 5, 4 >>> a array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19]]) >>> i = np.array([[2, 1], [3, 1], [1, 1], [0, 0], [3, 1]]) >>> timeit rowtake(a, i) 100000 loops, best of 3: 14.7 us per loop >>> timeit rowtake_cy(a, i) 100000 loops, best of 3: 10.6 us per loop So now it's almost as fast as the element-by-element Cython version. On 2010-08-06 03:29, josef.pktd at gmail.com wrote: > I still find broadcasting easier to read, even if it might be a bit slower > >>>> a[np.arange(5)[:,None], i] > array([[ 2, 1], > [ 7, 5], > [ 9, 9], > [12, 12], > [19, 17]]) Josef, I'd forgotten you could use None to increase the dimensionality of an array. Neat. And, somehow, it's almost twice as fast as the Cython version!: >>> timeit a[np.arange(a.shape[0])[:, None], i] 100000 loops, best of 3: 5.76 us per loop I like it. Thanks for all the help! Martin From n.becker at amolf.nl Fri Aug 6 16:53:58 2010 From: n.becker at amolf.nl (Nils Becker) Date: Fri, 06 Aug 2010 22:53:58 +0200 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: Message-ID: <4C5C7666.80703@amolf.nl> Hi again, first a correction: I posted > I believe np.histogram(data, bins, normed=True) effectively does : >>> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). >>> >>> However, it _should_ do >>> np.histogram(data, bins, normed=False) / bins_widths but there is a normalization missing; it should read I believe np.histogram(data, bins, normed=True) effectively does np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]) / data.sum() However, it _should_ do np.histogram(data, bins, normed=False) / bins_widths / data.sum() Bruce Southey replied: > As I recall, there as issues with this aspect. > Please search the discussion regarding histogram especially David > Huard's reply in this thread: > http://thread.gmane.org/gmane.comp.python.numeric.general/22445 I think this discussion pertains to a switch in calling conventions which happened at the time. The last reply of D. Huard (to me) seems to say that they did not fix anything in the _old_ semantics, but that the new semantics is expected to work properly. I tried with an infinite bin: counts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf]) counts array([1,3]) ncounts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf], normed=1) ncounts array([0.,0.]) this also does not make a lot of sense to me. A better result would be array([0.25, 0.]), since 25% of the points fall in the first bin; 75% fall in the second but are spread out over an infinite interval, giving 0. This is what my second proposal would give. I cannot find anything wrong with it so far... Cheers, Nils From numpy at mspacek.mm.st Fri Aug 6 17:39:41 2010 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 06 Aug 2010 14:39:41 -0700 Subject: [Numpy-discussion] use index array of len n to select columns of n x m array In-Reply-To: <4C5C6C8B.9030006@mspacek.mm.st> References: <4C5AF112.5090801@mspacek.mm.st> <4C5BDD66.9030002@mspacek.mm.st> <4C5C6C8B.9030006@mspacek.mm.st> Message-ID: <4C5C811D.1010205@mspacek.mm.st> On 2010-08-06 13:11, Martin Spacek wrote: > Josef, I'd forgotten you could use None to increase the dimensionality of an > array. Neat. And, somehow, it's almost twice as fast as the Cython version!: > > >>> timeit a[np.arange(a.shape[0])[:, None], i] > 100000 loops, best of 3: 5.76 us per loop I just realized why the Cython version was slower - the two assertion lines. commenting those out: @cython.boundscheck(False) @cython.wraparound(False) @cython.cdivision(True) # might be necessary to release the GIL? def rowtake_cy(np.ndarray[np.int32_t, ndim=2] a, np.ndarray[np.int32_t, ndim=2] i): """For each row in a, return values according to column indices in the corresponding row in i. Returned shape == i.shape""" cdef Py_ssize_t nrows, ncols, rowi, coli cdef np.ndarray[np.int32_t, ndim=2] out nrows = i.shape[0] ncols = i.shape[1] # num cols to take for each row #assert a.shape[0] == nrows #assert i.max() < a.shape[1] out = np.empty((nrows, ncols), dtype=np.int32) for rowi in range(nrows): for coli in range(ncols): out[rowi, coli] = a[rowi, i[rowi, coli]] return out gives me: >>> timeit rowtake_cy(a, i) 1000000 loops, best of 3: 1.44 us per loop which is 4X faster than the a[np.arange(a.shape[0])[:, None], i] broadcasting method. Martin From josef.pktd at gmail.com Fri Aug 6 19:49:45 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Aug 2010 19:49:45 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: <4C5C7666.80703@amolf.nl> References: <4C5C7666.80703@amolf.nl> Message-ID: On Fri, Aug 6, 2010 at 4:53 PM, Nils Becker wrote: > Hi again, > > first a correction: I posted > >> I believe np.histogram(data, bins, normed=True) effectively does : >>>> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). >>>> >>>> However, it _should_ do >>>> np.histogram(data, bins, normed=False) / bins_widths > > but there is a normalization missing; it should read > > I believe np.histogram(data, bins, normed=True) effectively does > np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]) / data.sum() > > However, it _should_ do > np.histogram(data, bins, normed=False) / bins_widths / data.sum() > > Bruce Southey replied: >> As I recall, there as issues with this aspect. >> Please search the discussion regarding histogram especially David >> Huard's reply in this thread: >> http://thread.gmane.org/gmane.comp.python.numeric.general/22445 > I think this discussion pertains to a switch in calling conventions > which happened at the time. The last reply of D. Huard (to me) seems to > say that they did not fix anything in the _old_ semantics, but that the > new semantics is expected to work properly. > > I tried with an infinite bin: > counts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf]) > counts > array([1,3]) > ncounts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf], normed=1) > ncounts > array([0.,0.]) > > this also does not make a lot of sense to me. A better result would be > array([0.25, 0.]), since 25% of the points fall in the first bin; 75% > fall in the second but are spread out over an infinite interval, giving > 0. This is what my second proposal would give. I cannot find anything > wrong with it so far... I didn't find any different information about the meaning of normed=True on the mailing list nor in the trac history 169 170 if normed: 171 db = array(np.diff(bins), float) 172 return n/(n*db).sum(), bins this does not look like the correct piecewise density with unequal binsizes. Thanks Nils for pointing this out, I tried only equal binsizes for a histogram distribution. Josef > > Cheers, Nils > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From oliphant at enthought.com Fri Aug 6 20:07:21 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 6 Aug 2010 19:07:21 -0500 Subject: [Numpy-discussion] dtype.type for structured arrays In-Reply-To: References: Message-ID: <96905C2D-8D7F-46BA-B0C0-C5A35946722B@enthought.com> On Jul 24, 2010, at 2:42 PM, Thomas Robitaille wrote: > Hi, > > If I create a structured array with vector columns: > >>>> array = np.array(zip([[1,2],[1,2],[1,3]]),dtype=[('a',float,2)]) > > then examine the type of the column, I get: > >>>> array.dtype[0] > dtype(('float64',(2,))) > > Then, if I try and view the numerical type, I see: > >>>> array.dtype[0].type > > > I have to basically do > >>>> array.dtype[0].subdtype[0] > dtype('float64') > > to get what I need. I seem to remember that this used not to be the case, and that even for vector columns, one could access array.dtype[0].type to get the numerical type. Is this a bug, or deliberate? > This looks the same as I remember it. The dtype is a structured array with the filed name 'a' having an element which is a vector of floats. As a result, the first dtype[0] extracts the vector of floats dtype. This must be type "void" because it is a vector of floats. To get to the underlying type, you have to do what you did. I don't see how it would have worked another way in the past. -Travis > Thanks, > > Thomas > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com From sturla at molden.no Sat Aug 7 15:37:17 2010 From: sturla at molden.no (Sturla Molden) Date: Sat, 7 Aug 2010 21:37:17 +0200 Subject: [Numpy-discussion] absoft and f2py Message-ID: <7b42c743f36f8a4b71e6da41fd0209b0.squirrel@webmail.uio.no> I asked before, but did not get an answer. What is the problem with absoft and f2py? Sturla C:\Absoft11.0>f2py -c --help-fcompiler Forcing DISTUTILS_USE_SDK=1 Unexpected ValueError in C:\Python26\lib\site-packages\numpy\distutils\fcompiler \compaq.py Traceback (most recent call last): File "C:\Python26\Scripts\f2py-script.py", line 24, in main() File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, in main run_compile() File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, in run_co mpile setup(ext_modules = [ext]) File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line 186, in set up return old_setup(**new_attr) File "C:\Python26\lib\distutils\core.py", line 138, in setup ok = dist.parse_command_line() File "C:\Python26\lib\distutils\dist.py", line 460, in parse_command_line args = self._parse_command_opts(parser, args) File "C:\Python26\lib\distutils\dist.py", line 574, in _parse_command_opts func() File "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py ", line 13, in show_fortran_compilers show_fcompilers(dist) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", li ne 848, in show_fcompilers load_all_fcompiler_classes() File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", li ne 715, in load_all_fcompiler_classes __import__ (module_name) File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\compaq.py", line 55, in class CompaqVisualFCompiler(FCompiler): File "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\compaq.py", line 95, in CompaqVisualFCompiler raise e ValueError: [u'path', u'lib'] From sturla at molden.no Sat Aug 7 16:07:38 2010 From: sturla at molden.no (Sturla Molden) Date: Sat, 7 Aug 2010 22:07:38 +0200 Subject: [Numpy-discussion] absoft and f2py In-Reply-To: <7b42c743f36f8a4b71e6da41fd0209b0.squirrel@webmail.uio.no> References: <7b42c743f36f8a4b71e6da41fd0209b0.squirrel@webmail.uio.no> Message-ID: <58091f3059dfac96d6e658e23593b7ff.squirrel@webmail.uio.no> After deleting compaq.py from C:\Python26\lib\site-packages\numpy\distutils\fcompiler, and removing compaq from _default_compilers in C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py, I get this: C:\Absoft11.0>f2py -c --help-fcompiler Forcing DISTUTILS_USE_SDK=1 AbsoftFCompiler instance properties: archiver = None compile_switch = '-c' compiler_f77 = ['C:\\Absoft11.0\\BIN\\f77.exe', '-N22', '-N90', '- N110', '-f', '-N15', '-O'] compiler_f90 = ['C:\\Absoft11.0\\BIN\\f90.exe', '-YCFRL=1', '- YCOM_NAMES=LCS', '-YCOM_PFX', '-YEXT_PFX', '-YCOM_SFX=_', '-YEXT_SFX=_', '-YEXT_NAMES=LCS', '-YDEALLOC=ALL', '-O'] compiler_fix = ['C:\\Absoft11.0\\BIN\\f90.exe', '-YCFRL=1', '- YCOM_NAMES=LCS', '-YCOM_PFX', '-YEXT_PFX', '-YCOM_SFX=_', '-YEXT_SFX=_', '-YEXT_NAMES=LCS', '-f', 'fixed', '- YCFRL=1', '-YCOM_NAMES=LCS', '-YCOM_PFX', '-YEXT_PFX', '- YCOM_SFX=_', '-YEXT_SFX=_', '-YEXT_NAMES=LCS', '- YDEALLOC=ALL', '-O'] libraries = ['af90math', 'afio', 'af77math', 'U77', 'COMDLG32'] library_dirs = ['C:\\Absoft11.0\\shlib64'] linker_exe = None linker_so = ['C:\\Absoft11.0\\BIN\\f90.exe', '/dll'] object_switch = '-o ' ranlib = None version = LooseVersion ('11.0') version_cmd = ['C:\\Absoft11.0\\BIN\\f90.exe', '-V', '-c', 'c:\\users\\sturla\\appdata\\local\\temp\\tmp8su73c\\rxhgr 8.f', '-o', 'c:\\users\\sturla\\appdata\\local\\temp\\tmp8su73c\\rxhgr 8.o'] Fortran compilers found: --fcompiler=absoft Absoft Corp Fortran Compiler (11.0) Compilers available for this platform, but not found: --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu GNU Fortran 77 compiler --fcompiler=gnu95 GNU Fortran 95 compiler --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps Compilers not available on this platform: --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun or Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler For compiler details, run 'config_fc --verbose' setup command. So it seems there is a bug in compaq.py that causes f2py to crash. I am running Enthought 6.2-2 on amd64. (Which means I have NumPy 1.4.0.) Regards, Sturla Molden > > I asked before, but did not get an answer. What is the problem with absoft > and f2py? > > Sturla > > > > C:\Absoft11.0>f2py -c --help-fcompiler > Forcing DISTUTILS_USE_SDK=1 > Unexpected ValueError in > C:\Python26\lib\site-packages\numpy\distutils\fcompiler > \compaq.py > Traceback (most recent call last): > File "C:\Python26\Scripts\f2py-script.py", line 24, in > main() > File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 557, in > main > run_compile() > File "C:\Python26\lib\site-packages\numpy\f2py\f2py2e.py", line 543, in > run_co > mpile > setup(ext_modules = [ext]) > File "C:\Python26\lib\site-packages\numpy\distutils\core.py", line 186, > in set > up > return old_setup(**new_attr) > File "C:\Python26\lib\distutils\core.py", line 138, in setup > ok = dist.parse_command_line() > File "C:\Python26\lib\distutils\dist.py", line 460, in > parse_command_line > args = self._parse_command_opts(parser, args) > File "C:\Python26\lib\distutils\dist.py", line 574, in > _parse_command_opts > func() > File > "C:\Python26\lib\site-packages\numpy\distutils\command\config_compiler.py > ", line 13, in show_fortran_compilers > show_fcompilers(dist) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", > li > ne 848, in show_fcompilers > load_all_fcompiler_classes() > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\__init__.py", > li > ne 715, in load_all_fcompiler_classes > __import__ (module_name) > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\compaq.py", > line > 55, in > class CompaqVisualFCompiler(FCompiler): > File > "C:\Python26\lib\site-packages\numpy\distutils\fcompiler\compaq.py", > line > 95, in CompaqVisualFCompiler > raise e > ValueError: [u'path', u'lib'] > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwmsmith at gmail.com Sat Aug 7 20:33:39 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Sat, 7 Aug 2010 19:33:39 -0500 Subject: [Numpy-discussion] ANN fwrap v0.1.0 Message-ID: Fwrap v0.1.0 ============ I am pleased to announce the first release of Fwrap v0.1.0, a utility for wrapping Fortran code in C, Cython and Python. Fwrap focuses on supporting the features of Fortran 90/95, but will work with most well-behaved Fortran 77 code, too. Fwrap is BSD licensed. Fwrap is beta-level software; all public APIs and commandline options are subject to change. Features in the v0.1.0 release: * Fortran source parsing and automatic wrapper generation; * Top-level (non-module) Fortran subroutines and functions; * Supports all intrinsic types (integer, real, complex, logical & character); * Default and non-default intrinsic types properly wrapped; * Scalar and array arguments supported; * Supports assumed-shape, assumed-size, and explicit-shape array arguments; * Intent 'in', 'inout', 'out' and no intent arguments supported; * Automatic docstring generation for extension module and functions; * Wrappers are portable w.r.t. both Fortran and C compilers. Upcoming features: * "Use" statement support; * Python & Cython callback support; * Kind-parameters in type specification; * Module-level data and parameters; * User-derived types. See the README and documentation for requirements, compiler support, etc. Download -------- You can get fwrap from pypi or from its sourceforge download page: https://sourceforge.net/projects/fwrap/files/ More Info --------- Project homepage, including links to wiki & bug tracker: http://fwrap.sourceforge.net/ Mailing list: http://groups.google.com/group/fwrap-users Development blog: http://fortrancython.wordpress.com/ From robertwb at math.washington.edu Sun Aug 8 00:12:20 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Aug 2010 21:12:20 -0700 Subject: [Numpy-discussion] [cython-users] ANN fwrap v0.1.0 In-Reply-To: References: Message-ID: Excellent news! May you get lots of users and (easy to address) bug reports. Sounds like it does a lot already. - Robert On Sat, Aug 7, 2010 at 5:33 PM, Kurt Smith wrote: > Fwrap v0.1.0 > ============ > > I am pleased to announce the first release of Fwrap v0.1.0, a utility for > wrapping Fortran code in C, Cython and Python. ?Fwrap focuses on supporting the > features of Fortran 90/95, but will work with most well-behaved Fortran 77 > code, too. > > Fwrap is BSD licensed. > > Fwrap is beta-level software; all public APIs and commandline options are > subject to change. > > Features in the v0.1.0 release: > > ? ?* Fortran source parsing and automatic wrapper generation; > > ? ?* Top-level (non-module) Fortran subroutines and functions; > > ? ?* Supports all intrinsic types (integer, real, complex, logical & > ? ? ?character); > > ? ?* Default and non-default intrinsic types properly wrapped; > > ? ?* Scalar and array arguments supported; > > ? ?* Supports assumed-shape, assumed-size, and explicit-shape array arguments; > > ? ?* Intent 'in', 'inout', 'out' and no intent arguments supported; > > ? ?* Automatic docstring generation for extension module and functions; > > ? ?* Wrappers are portable w.r.t. both Fortran and C compilers. > > Upcoming features: > > ? ?* "Use" statement support; > > ? ?* Python & Cython callback support; > > ? ?* Kind-parameters in type specification; > > ? ?* Module-level data and parameters; > > ? ?* User-derived types. > > See the README and documentation for requirements, compiler support, etc. > > Download > -------- > > You can get fwrap from pypi or from its sourceforge download page: > > ? ?https://sourceforge.net/projects/fwrap/files/ > > More Info > --------- > > Project homepage, including links to wiki & bug tracker: > > ? ?http://fwrap.sourceforge.net/ > > Mailing list: > > ? ?http://groups.google.com/group/fwrap-users > > Development blog: > > ? ?http://fortrancython.wordpress.com/ > From ralf.gommers at googlemail.com Sun Aug 8 03:05:55 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 8 Aug 2010 15:05:55 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: <4FE4E2B9-7196-4356-8213-8358C783BF78@cs.toronto.edu> References: <4FE4E2B9-7196-4356-8213-8358C783BF78@cs.toronto.edu> Message-ID: On Fri, Aug 6, 2010 at 6:05 AM, David Warde-Farley wrote: > > On 2010-08-01, at 12:38 PM, Ralf Gommers wrote: > > I am pleased to announce the availability of the first beta of NumPy 1.5.0. > This will be the first NumPy release to include support for Python 3, as > well as for Python 2.7. Please try this beta and report any problems on the > NumPy mailing list. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > Please note that binaries for Python 3.1 are not yet up, they will follow > as soon as a minor issue with building them is resolved. Building from > source with Python 3.1 should work without problems. > > > Hey Ralf, > > I am getting a single test failure: > > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/dwf/pkg/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 275, in test_special_values > assert_almost_equal(np.log(np.conj(xa[i])), np.conj(np.log(xa[i]))) > File "/home/dwf/pkg/lib/python2.7/site-packages/numpy/testing/utils.py", > line 443, in assert_almost_equal > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal > ACTUAL: array([-inf+3.14159265j]) > DESIRED: array([-inf-3.14159265j]) > >> raise AssertionError('\nArrays are not almost equal\n ACTUAL: > array([-inf+3.14159265j])\n DESIRED: array([-inf-3.14159265j])') > > This is on a Xeon E5540 built against the MKL 11.1, if that matters (I > suspect it doesn't). > > May be related to the recent inf/nan changes, but this test has been failing for a long time on Windows and was marked as knownfail in 1.4.1. It doesn't look like a serious problem, so we may just mark it knownfail again. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Aug 8 03:09:00 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 8 Aug 2010 15:09:00 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: <4C562DC7.3070109@gmail.com> References: <4C55FEE3.40303@gmail.com> <4C562DC7.3070109@gmail.com> Message-ID: On Mon, Aug 2, 2010 at 10:30 AM, Alan G Isaac wrote: > Tests produce a few failures and a couple warnings. > Details below. > Alan Isaac > > > Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] > on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > >>> np.test() > Running unit tests for numpy > NumPy version 1.5.0b1 > NumPy is installed in C:\Python27\lib\site-packages\numpy > Python version 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit > (Intel)] > nose version 0.11.0 > > .......................................................................................... > > ....................E..................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .....................K.................................................................... > > .......................................................................................... > > ..................................................................K....................... > ...................................K..K.............Warning: invalid value > encountered in > absolute > .Warning: invalid value encountered in absolute > > ................K........F....FFFF.....K.F.........FF..FFF.....F.......................... > > ......................................................S................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................K.........K........................................ > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > ..........................................S............................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > > .......................................................................................... > ................................................................ > ====================================================================== > ERROR: test_filename (test_memmap.TestMemmap) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\numpy\core\tests\test_memmap.py", > line 60, in test_f > ilename > os.unlink(tmpname) > WindowsError: [Error 32] The process cannot access the file because it is > being used by an > other process: 'c:\\users\\alanis~1\\appdata\\local\\temp\\mmapoo5c2b' > > This should be fixed in r8606. > ====================================================================== > FAIL: test_umath_complex.TestCexp.test_special_values(, inf, > 0, inf, 0) > test_umath_complex.TestCexp.test_special_values(, inf, 0, inf, > 0) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 539, i > n check_complex_value > assert_equal(f(z1), z2) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, > in assert_array_e > qual > verbose=verbose, header='Arrays are not equal') > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, > in assert_array_c > ompare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (x and y nan location mismatch [ True], False mismatch) > x: array([ inf nanj]) > y: array((inf+0j)) > >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location > mismatch [ True] > , False mismatch)\n x: array([ inf nanj])\n y: array((inf+0j))') > > > ====================================================================== > FAIL: test_umath_complex.TestCexp.test_special_values(None,) > test_umath_complex.TestCexp.test_special_values(None,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 60, in > _check_ninf_inf > raise AssertionError(msgform %(z.real, z.imag)) > AssertionError: cexp(-inf, inf) is (nan, nan), expected (+-0, +-0) > >> raise AssertionError('cexp(-inf, inf) is (%f, %f), expected (+-0, +-0)' > %((nan+nan*j). > real, (nan+nan*j).imag)) > > > ====================================================================== > FAIL: test_umath_complex.TestCexp.test_special_values(None,) > test_umath_complex.TestCexp.test_special_values(None,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 73, in > _check_inf_inf > raise AssertionError(msgform % (z.real, z.imag)) > AssertionError: cexp(inf, inf) is (nan, nan), expected (+-inf, nan) > >> raise AssertionError('cexp(inf, inf) is (%f, %f), expected (+-inf, > nan)' % ((nan+nan*j > ).real, (nan+nan*j).imag)) > > > ====================================================================== > FAIL: test_umath_complex.TestCexp.test_special_values(None,) > test_umath_complex.TestCexp.test_special_values(None,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 86, in > _check_ninf_nan > raise AssertionError(msgform % (z.real, z.imag)) > AssertionError: cexp(-inf, nan) is (nan, nan), expected (+-0, +-0) > >> raise AssertionError('cexp(-inf, nan) is (%f, %f), expected (+-0, +-0)' > % ((nan+nan*j) > .real, (nan+nan*j).imag)) > > > ====================================================================== > FAIL: test_umath_complex.TestCexp.test_special_values(None,) > test_umath_complex.TestCexp.test_special_values(None,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 99, in > _check_inf_nan > raise AssertionError(msgform % (z.real, z.imag)) > AssertionError: cexp(-inf, nan) is (nan, nan), expected (+-inf, nan) > >> raise AssertionError('cexp(-inf, nan) is (%f, %f), expected (+-inf, > nan)' % ((nan+nan* > j).real, (nan+nan*j).imag)) > > > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 144, i > n test_special_values > self.assertRaises(FloatingPointError, np.log, x) > AssertionError: FloatingPointError not raised > > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, > inf, inf, inf) > test_umath_complex.TestCsqrt.test_special_values(, 1, inf, > inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 539, i > n check_complex_value > assert_equal(f(z1), z2) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, > in assert_array_e > qual > verbose=verbose, header='Arrays are not equal') > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, > in assert_array_c > ompare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (x and y nan location mismatch [ True], False mismatch) > x: array([ inf nanj]) > y: array((inf+infj)) > >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location > mismatch [ True] > , False mismatch)\n x: array([ inf nanj])\n y: array((inf+infj))') > > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, > inf, inf, inf) > test_umath_complex.TestCsqrt.test_special_values(, -1, inf, > inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 539, i > n check_complex_value > assert_equal(f(z1), z2) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, > in assert_array_e > qual > verbose=verbose, header='Arrays are not equal') > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, > in assert_array_c > ompare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (x and y nan location mismatch [ True], False mismatch) > x: array([ nan infj]) > y: array((inf+infj)) > >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location > mismatch [ True] > , False mismatch)\n x: array([ nan infj])\n y: array((inf+infj))') > > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, > inf, inf, inf) > > test_umath_complex.TestCsqrt.test_special_values(, inf, inf, > inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 539, i > n check_complex_value > assert_equal(f(z1), z2) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, > in assert_array_e > qual > verbose=verbose, header='Arrays are not equal') > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, > in assert_array_c > ompare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (x and y nan location mismatch [ True], False mismatch) > x: array([ inf nanj]) > y: array((inf+infj)) > >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location > mismatch [ True] > , False mismatch)\n x: array([ inf nanj])\n y: array((inf+infj))') > > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > -inf, inf, inf, inf > ) > test_umath_complex.TestCsqrt.test_special_values(, -inf, inf, > inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 539, i > n check_complex_value > assert_equal(f(z1), z2) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, > in assert_array_e > qual > verbose=verbose, header='Arrays are not equal') > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, > in assert_array_c > ompare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (x and y nan location mismatch [ True], False mismatch) > x: array([ nan infj]) > y: array((inf+infj)) > >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location > mismatch [ True] > , False mismatch)\n x: array([ nan infj])\n y: array((inf+infj))') > > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, > inf, inf, inf) > > test_umath_complex.TestCsqrt.test_special_values(, nan, inf, > inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 539, i > n check_complex_value > assert_equal(f(z1), z2) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 686, > in assert_array_e > qual > verbose=verbose, header='Arrays are not equal') > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 596, > in assert_array_c > ompare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (x and y nan location mismatch [ True], False mismatch) > x: array([ nan nanj]) > y: array((inf+infj)) > >> raise AssertionError('\nArrays are not equal\n\n(x and y nan location > mismatch [ True] > , False mismatch)\n x: array([ nan nanj])\n y: array((inf+infj))') > > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(None,) > test_umath_complex.TestCsqrt.test_special_values(None,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 183, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line > 340, i > n _check_ninf_nan > raise AssertionError(msgform % (z.real, z.imag)) > AssertionError: csqrt(-inf, nan) is (nan, nan), expected (nan, +-inf) > >> raise AssertionError('csqrt(-inf, nan) is (%f, %f), expected (nan, > +-inf)' % ((nan+nan > *j).real, (nan+nan*j).imag)) > > > Pauli or Charles, can you please have a look at these? Looks like that's related to your recent work on nans/infs. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Aug 8 07:20:52 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 8 Aug 2010 11:20:52 +0000 (UTC) Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 References: <4C55FEE3.40303@gmail.com> <4C562DC7.3070109@gmail.com> Message-ID: Sun, 08 Aug 2010 15:09:00 +0800, Ralf Gommers wrote: > On Mon, Aug 2, 2010 at 10:30 AM, Alan G Isaac [clip] >> ====================================================================== >> FAIL: test_umath_complex.TestCsqrt.test_special_values(, >> 1, inf, inf, inf) >> test_umath_complex.TestCsqrt.test_special_values(, 1, >> inf, inf, inf) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Python27\lib\site-packages\nose\case.py", line 183, in >> runTest >> self.test(*self.arg) >> File >> "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", >> line 539, i >> n check_complex_value >> assert_equal(f(z1), z2) >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 256, >> in assert_equal >> return assert_array_equal(actual, desired, err_msg, verbose) >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 686, >> in assert_array_e >> qual >> verbose=verbose, header='Arrays are not equal') >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line >> 596, >> in assert_array_c >> ompare >> raise AssertionError(msg) >> AssertionError: >> Arrays are not equal >> >> (x and y nan location mismatch [ True], False mismatch) >> x: array([ inf nanj]) >> y: array((inf+infj)) >> >> raise AssertionError('\nArrays are not equal\n\n(x and y nan >> >> location >> mismatch [ True] >> , False mismatch)\n x: array([ inf nanj])\n y: array((inf+infj))') [clip] > Pauli or Charles, can you please have a look at these? Looks like > that's related to your recent work on nans/infs. These errors probably come from the fact that the platform's C library does not handle special nan/inf values in csqrt in the way C99 standard requires. The recent changes in umath since 1.4.x are unrelated. I think the support for platform-provided complex functions was enabled in r7986. To be sure, we'd need to see the build log. -- Pauli Virtanen From ralf.gommers at googlemail.com Sun Aug 8 11:33:26 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 8 Aug 2010 23:33:26 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: <4C55FEE3.40303@gmail.com> <4C562DC7.3070109@gmail.com> Message-ID: On Sun, Aug 8, 2010 at 7:20 PM, Pauli Virtanen wrote: > Sun, 08 Aug 2010 15:09:00 +0800, Ralf Gommers wrote: > > On Mon, Aug 2, 2010 at 10:30 AM, Alan G Isaac > [clip] > >> ====================================================================== > >> FAIL: test_umath_complex.TestCsqrt.test_special_values(, > >> 1, inf, inf, inf) > >> test_umath_complex.TestCsqrt.test_special_values(, 1, > >> inf, inf, inf) > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Python27\lib\site-packages\nose\case.py", line 183, in > >> runTest > >> self.test(*self.arg) > >> File > >> "C:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", > >> line 539, i > >> n check_complex_value > >> assert_equal(f(z1), z2) > >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > >> 256, > >> in assert_equal > >> return assert_array_equal(actual, desired, err_msg, verbose) > >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > >> 686, > >> in assert_array_e > >> qual > >> verbose=verbose, header='Arrays are not equal') > >> File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line > >> 596, > >> in assert_array_c > >> ompare > >> raise AssertionError(msg) > >> AssertionError: > >> Arrays are not equal > >> > >> (x and y nan location mismatch [ True], False mismatch) > >> x: array([ inf nanj]) > >> y: array((inf+infj)) > >> >> raise AssertionError('\nArrays are not equal\n\n(x and y nan > >> >> location > >> mismatch [ True] > >> , False mismatch)\n x: array([ inf nanj])\n y: array((inf+infj))') > [clip] > > Pauli or Charles, can you please have a look at these? Looks like > > that's related to your recent work on nans/infs. > > These errors probably come from the fact that the platform's C library > does not handle special nan/inf values in csqrt in the way C99 standard > requires. The recent changes in umath since 1.4.x are unrelated. > > I think the support for platform-provided complex functions was enabled > in r7986. > > To be sure, we'd need to see the build log. > > Ticket opened with 2.7 and 2.6 build logs attached, http://projects.scipy.org/numpy/ticket/1574. Looks like it tries to use C99: #define NPY_USE_C99_COMPLEX #define NPY_HAVE_COMPLEX_DOUBLE 1 #define NPY_HAVE_COMPLEX_FLOAT 1 #define NPY_HAVE_COMPLEX_LONG_DOUBLE 1 #define NPY_USE_C99_FORMATS 1 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdmcbain at freeshell.org Sun Aug 8 22:37:17 2010 From: gdmcbain at freeshell.org (Geordie McBain) Date: Mon, 9 Aug 2010 12:37:17 +1000 Subject: [Numpy-discussion] numpy.ma.flatnotmasked_contiguous error in slice.stop values? Message-ID: The function numpy.ma.flatnotmasked_contiguous returns slices which miss the last element; for example, based on the example at http://docs.scipy.org/doc/numpy/reference/generated/numpy.ma.flatnotmasked_contiguous.html, import numpy as np a = np.arange (10) am = np.ma.array (a, mask = (a < 3) | (a > 8) | (a == 5)) segments = np.ma.flatnotmasked_contiguous (am) print 'the masked array:', am print 'flatnotmasked_contiguous: ', segments print am[segments[0]], am[segments[1]] prints the masked array: [-- -- -- 3 4 -- 6 7 8 --] flatnotmasked_contiguous: [slice(3, 4, None), slice(6, 8, None)] [3] [6 7] which isn't really what we're after; I would have expected [3 4] and [6 7 8]. Compare the similarly named matplotlib.mlab.contiguous_regions which does have the expected behaviour (although its output format is different) import matplotlib.mlab segments1 = matplotlib.mlab.contiguous_regions (am) print 'contiguous_regions:', segments1 print am[slice (*segments1[0])], am[slice (*segments1[1])] which prints contiguous_regions: [(3, 5), (6, 9)] [3 4] [6 7 8] (I'm running Python 2.6.4 on 64-bit Linux Mint 8 `Helena', with NumPy from Ubuntu Karmic, which I think means I have NumPy version 1.3.0?) From pgmdevlist at gmail.com Sun Aug 8 22:47:00 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Sun, 8 Aug 2010 22:47:00 -0400 Subject: [Numpy-discussion] numpy.ma.flatnotmasked_contiguous error in slice.stop values? In-Reply-To: References: Message-ID: <068D97B1-FF8E-41A5-A84D-C2E0C1465436@gmail.com> On Aug 8, 2010, at 10:37 PM, Geordie McBain wrote: > The function numpy.ma.flatnotmasked_contiguous returns slices which > miss the last element; for example, based on the example at > http://docs.scipy.org/doc/numpy/reference/generated/numpy.ma.flatnotmasked_contiguous.html, OK, mind opening a ticket ? I'll try to take care of that tmw and I don't want the bug report to fall between the cracks. > > which isn't really what we're after; I would have expected [3 4] and > [6 7 8]. Compare the similarly named > matplotlib.mlab.contiguous_regions which does have the expected > behaviour (although its output format is different) Oh, I didn't know there was such an option in mpl. I wonder when it got introduced... > import matplotlib.mlab > segments1 = matplotlib.mlab.contiguous_regions (am) > print 'contiguous_regions:', segments1 > print am[slice (*segments1[0])], am[slice (*segments1[1])] > > which prints > > contiguous_regions: [(3, 5), (6, 9)] Mmh, tuples are nicer than slices... We'll see. > [3 4] [6 7 8] > > (I'm running Python 2.6.4 on 64-bit Linux Mint 8 `Helena', with NumPy > from Ubuntu Karmic, which I think means I have NumPy version 1.3.0?) print numpy.version.version From gdmcbain at freeshell.org Sun Aug 8 23:54:32 2010 From: gdmcbain at freeshell.org (Geordie McBain) Date: Mon, 9 Aug 2010 13:54:32 +1000 Subject: [Numpy-discussion] numpy.ma.flatnotmasked_contiguous error in slice.stop values? In-Reply-To: <068D97B1-FF8E-41A5-A84D-C2E0C1465436@gmail.com> References: <068D97B1-FF8E-41A5-A84D-C2E0C1465436@gmail.com> Message-ID: 2010/8/9 Pierre GM : > OK, mind opening a ticket ? I'll try to take care of that tmw and I don't want the bug report to fall between the cracks. Done. http://projects.scipy.org/numpy/ticket/1576. Thank you for your very prompt and helpful response. I wasn't aware of the ticket system previously. > Mmh, tuples are nicer than slices... We'll see. O.K. I'm fairly new to NumPy and Matplotlib, so I'll leave the design decision to the experts, but I did find segments = np.ma.flatnotmasked_contiguous (am) print am[segments[0]], am[segments[1]] easier than segments1 = matplotlib.mlab.contiguous_regions (am) print am[slice (*segments1[0])], am[slice (*segments1[1])] but perhaps there are more elegant ways of dealing with tuples representing slices? > print numpy.version.version Thanks. From charlesr.harris at gmail.com Tue Aug 10 21:54:54 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 10 Aug 2010 19:54:54 -0600 Subject: [Numpy-discussion] numpy.ma.flatnotmasked_contiguous error in slice.stop values? In-Reply-To: <068D97B1-FF8E-41A5-A84D-C2E0C1465436@gmail.com> References: <068D97B1-FF8E-41A5-A84D-C2E0C1465436@gmail.com> Message-ID: Hi Pierre, On Sun, Aug 8, 2010 at 8:47 PM, Pierre GM wrote: > > On Aug 8, 2010, at 10:37 PM, Geordie McBain wrote: > > > The function numpy.ma.flatnotmasked_contiguous returns slices which > > miss the last element; for example, based on the example at > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.ma.flatnotmasked_contiguous.html > , > > OK, mind opening a ticket ? I'll try to take care of that tmw and I don't > want the bug report to fall between the cracks. > > > > which isn't really what we're after; I would have expected [3 4] and > > [6 7 8]. Compare the similarly named > > matplotlib.mlab.contiguous_regions which does have the expected > > behaviour (although its output format is different) > > Oh, I didn't know there was such an option in mpl. I wonder when it got > introduced... > > > import matplotlib.mlab > > segments1 = matplotlib.mlab.contiguous_regions (am) > > print 'contiguous_regions:', segments1 > > print am[slice (*segments1[0])], am[slice (*segments1[1])] > > > > which prints > > > > contiguous_regions: [(3, 5), (6, 9)] > > Mmh, tuples are nicer than slices... We'll see. > > > [3 4] [6 7 8] > > > > (I'm running Python 2.6.4 on 64-bit Linux Mint 8 `Helena', with NumPy > > from Ubuntu Karmic, which I think means I have NumPy version 1.3.0?) > > print numpy.version.version > > I'm getting these tests errors now. ====================================================================== FAIL: Test flatnotmasked_contiguous ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/numpy/ma/tests/test_extras.py", line 104, in test_flatnotmasked_contiguous assert_equal(test, []) File "/usr/local/lib/python2.6/dist-packages/numpy/ma/testutils.py", line 98, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: None DESIRED: [] >> raise AssertionError('\nItems are not equal:\n ACTUAL: None\n DESIRED: []') ====================================================================== FAIL: Tests unmasked_edges ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/numpy/ma/tests/test_extras.py", line 254, in test_edges assert_equal(test, [0, -1]) File "/usr/local/lib/python2.6/dist-packages/numpy/ma/testutils.py", line 121, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/usr/local/lib/python2.6/dist-packages/numpy/ma/testutils.py", line 193, in assert_array_equal header='Arrays are not equal') File "/usr/local/lib/python2.6/dist-packages/numpy/ma/testutils.py", line 186, in assert_array_compare verbose=verbose, header=header) File "/usr/local/lib/python2.6/dist-packages/numpy/testing/utils.py", line 618, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 50.0%) x: array([ 0, 24]) y: array([ 0, -1]) >> raise AssertionError('\nArrays are not equal\n\n(mismatch 50.0%)\n x: array([ 0, 24])\n y: array([ 0, -1])') Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue Aug 10 22:02:31 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 10 Aug 2010 22:02:31 -0400 Subject: [Numpy-discussion] numpy.ma.flatnotmasked_contiguous error in slice.stop values? In-Reply-To: References: <068D97B1-FF8E-41A5-A84D-C2E0C1465436@gmail.com> Message-ID: On Aug 10, 2010, at 9:54 PM, Charles R Harris wrote: > Hi Pierre, > > I'm getting these tests errors now. I forgot to commit some last minute modifications... And now (r8621) ? Thinking about it, the function `flatnotmasked_contiguous` outputs a list of slices, except when there's none (and then it outputs None). Shouldn't it output an empty list instead ? From renato.fabbri at gmail.com Wed Aug 11 11:00:22 2010 From: renato.fabbri at gmail.com (Renato Fabbri) Date: Wed, 11 Aug 2010 12:00:22 -0300 Subject: [Numpy-discussion] how to fit a given pdf Message-ID: Dear All, help appreciated, thanks in advance. how do you fit a pdf you have with a given pdf (say gamma). with the file attached, you can go like: a=open("AC-010_ED-1m37F100P0.txt","rb") aa=a.read() aaa=aa[1:-1].split(",") data=[int(i) for i in aaa] if you do pylab.plot(data); pylab.show() The data is something like: ___|\___ It is my pdf (probability density function). how can i find the right parameters to make that fit with a gamma? if i was looking for a normal pdf, for example, i would just find mean and std and ask for the pdf. i've been playing with scipy.stats.distributions.gamma but i have not reached anything. we can extend the discussion further, but this is a good starting point. any idea? thanks! best of luck, rf -- GNU/Linux User #479299 skype: fabbri.renato -------------- next part -------------- [3, 1, 3, 2, 2, 2, 1, 1, 1, 1, 5, 4, 9, 16, 26, 51, 55, 83, 134, 228, 222, 202, 203, 180, 142, 114, 75, 55, 48, 41, 28, 26, 17, 13, 11, 14, 15, 14, 16, 9, 3, 3, 4, 3, 7, 4, 3, 3, 5, 7, 2, 2, 1, 8, 10, 14, 22, 11, 10, 12, 6, 3, 2, 1] From jsalvati at u.washington.edu Wed Aug 11 16:24:48 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Wed, 11 Aug 2010 13:24:48 -0700 Subject: [Numpy-discussion] Numpy datetime support Message-ID: Hello, I am an eager for numpy datetime support, but I have not heard about it recently. Last I heard it was taken out of 1.4 because of some compatibility issues. When will datetime be a full numpy feature? Best Regards, John -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdmcbain at freeshell.org Wed Aug 11 20:00:54 2010 From: gdmcbain at freeshell.org (Geordie McBain) Date: Thu, 12 Aug 2010 10:00:54 +1000 Subject: [Numpy-discussion] how to fit a given pdf In-Reply-To: References: Message-ID: 2010/8/12 Renato Fabbri : > Dear All, > > help appreciated, thanks in advance. > > how do you fit a pdf you have with a given pdf (say gamma). > > with the file attached, you can go like: > > a=open("AC-010_ED-1m37F100P0.txt","rb") > aa=a.read() > aaa=aa[1:-1].split(",") > data=[int(i) for i in aaa] > > if you do pylab.plot(data); pylab.show() > > The data is something like: > ___|\___ > > It is my pdf (probability density function). > > how can i find the right parameters to make that fit with a gamma? > > if i was looking for a normal pdf, for example, i would just find mean > and std and ask for the pdf. > > i've been playing with scipy.stats.distributions.gamma but i have not > reached anything. > > we can extend the discussion further, but this is a good starting point. > > any idea? A general point on fitting empirical probability density functions is that it is often much easier to fit the cumulative distribution function instead. For one thing, this means you don't have to decide on the intervals of the bins in the histogram. For another, it's actually often the cdf that is more related to the final answer (though I don't know your application, of course). Here's a quote. `So far the discussion of plots of distributions has emphasized frequency (or probability) vs. size plots, whereas for many applications cumulative plots are more important. Cumulative curves are produced by plotting the percentage of particles (or weight, volume, or surface) having particle diameters greater than (or less than) a given particle size against the particle size. ? Such curves have the advantage over histograms for plotting data that the class interval is eliminated, and they can be used to represent data which are obtained in classified form having unequal class intervals' (Cadle, R. D. 1965. Particle Size. New York: Reinhold Publishing Corporation, pp. 38-39) Once you've got your empirical cdf, the problem reduces to one of nonlinear curve fitting, for whichever theoretical distribution you like. For a tutorial on nonlinear curve fitting, see scipy.optimize.leastsq at http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html. You could of course use this approach for the pdf too, but I fancy the cdf result will be more robust. On the other hand, if you want something like your `mean and variance' approach to fitting normal distributions, you could still compare your mean and variance with the known values for the Gamma distribution (available e.g. on its Wikipedia page) and back-out the two parameters of the distribution from them. I'm not too sure how well this will work, but it's pretty easy. Another idea occurs to me and is about as easy as this is to compute the two parameters of the Gamma distribution by collocation with the empirical cdf; i.e. pick two quantiles, e.g. 0.25 and 0.75, or whatever, and get two equations for the two unknown parameters by insisting on the Gamma cdf agreeing with the empirical for these quantiles. This might be more robust than the mean & variance approach, but I haven't tried either. Good luck! From josef.pktd at gmail.com Wed Aug 11 21:51:04 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 11 Aug 2010 21:51:04 -0400 Subject: [Numpy-discussion] how to fit a given pdf In-Reply-To: References: Message-ID: On Wed, Aug 11, 2010 at 8:00 PM, Geordie McBain wrote: > 2010/8/12 Renato Fabbri : >> Dear All, >> >> help appreciated, thanks in advance. >> >> how do you fit a pdf you have with a given pdf (say gamma). >> >> with the file attached, you can go like: >> >> a=open("AC-010_ED-1m37F100P0.txt","rb") >> aa=a.read() >> aaa=aa[1:-1].split(",") >> data=[int(i) for i in aaa] >> >> if you do pylab.plot(data); pylab.show() >> >> The data is something like: >> ___|\___ >> >> It is my pdf (probability density function). >> >> how can i find the right parameters to make that fit with a gamma? >> >> if i was looking for a normal pdf, for example, i would just find mean >> and std and ask for the pdf. >> >> i've been playing with scipy.stats.distributions.gamma but i have not >> reached anything. >> >> we can extend the discussion further, but this is a good starting point. >> >> any idea? > > A general point on fitting empirical probability density functions is > that it is often much easier to fit the cumulative distribution > function instead. ?For one thing, this means you don't have to decide > on the intervals of the bins in the histogram. ?For another, it's > actually often the cdf that is more related to the final answer > (though I don't know your application, of course). > > Here's a quote. > > ?`So far the discussion of plots of distributions has emphasized > frequency (or probability) vs. size plots, whereas for many > applications cumulative plots are more important. Cumulative curves > are produced by plotting the percentage of particles (or weight, > volume, or surface) having particle diameters greater than (or less > than) a given particle size against the particle size. ? Such curves > have the advantage over histograms for plotting data that the class > interval is eliminated, and they can be used to represent data which > are obtained in classified form having unequal class intervals' > (Cadle, R. D. 1965. Particle Size. New York: Reinhold Publishing > Corporation, pp. 38-39) > > Once you've got your empirical cdf, the problem reduces to one of > nonlinear curve fitting, for whichever theoretical distribution you > like. ?For a tutorial on nonlinear curve fitting, see > scipy.optimize.leastsq at > http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html. ?You > could of course use this approach for the pdf too, but I fancy the cdf > result will be more robust. A similar approach for pareto works easily because the function is linear in log-log scale, but the properties of these graphical estimators was criticized in the readings as not being as good as other estimators. But they also show that in the graphs, empirical cdf or survival function are more stable/smooth than histograms, especially in the tails. MLE uses the pdf or logpdf without binning the data. > > On the other hand, if you want something like your `mean and variance' > approach to fitting normal distributions, you could still compare your > mean and variance with the known values for the Gamma distribution > (available e.g. on its Wikipedia page) and back-out the two parameters > of the distribution from them. ?I'm not too sure how well this will > work, but it's pretty easy. called method of moments in the literature, as in Jonathan's estimator > > Another idea occurs to me and is about as easy as this is to compute > the two parameters of the Gamma distribution by collocation with the > empirical cdf; i.e. pick two quantiles, e.g. 0.25 and 0.75, or > whatever, and get two equations for the two unknown parameters by > insisting on the Gamma cdf agreeing with the empirical for these > quantiles. ?This might be more robust than the mean & variance > approach, but I haven't tried either. It works pretty well, see http://www.johndcook.com/blog/2010/01/31/parameters-from-percentiles/ and his other articles cited at the bottom. (Using a generalized method of moment approach, this can be generalized to using more than two quantiles, which works well in my examples.) On advantage of generic MLE is that I know how to get (asymptotic) standard errors and confidence intervals, while for many of the other estimators bootstrap or checking the distribution specific formulas is necessary. Josef > > Good luck! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Luke.Pinner at environment.gov.au Thu Aug 12 01:19:43 2010 From: Luke.Pinner at environment.gov.au (Pinner, Luke) Date: Thu, 12 Aug 2010 15:19:43 +1000 Subject: [Numpy-discussion] Replacing values based on index Message-ID: <64102B2FB850C345B04D09C82018F45F0273EDBA@ACT01VEXCCL04.internal.govt> Apologies if this has been asked many times before, but I'm having trouble even coming up with the right terms to search on. I have an x,y,date stack of satellite images from which I need to know on what date the max and min values occur. I can calculate the index at which the min/max values occur, but how do I join that back to the date? A simple example case: import numpy as np #Date when image was captured, #images are in the stack in this order days=[10, 20, 31, 41, 51, 59, 69, 79, 90, 100] #dummy array representing something like #"stackofimages.argmin(axis=2)" indexmin=np.array([[0, 2, 1], [6, 7, 3], [5, 5, 8]]) #I want to convert indexmin to "daymin" i.e #[[10, 31, 20], # [69, 79, 41], # [59, 59, 90]] #Where the each element of indexmin represent the value #at the corresponding index in the days list/array. #What I am currently doing is below, #which is _rather_ slow on large arrays #as I'm calling np.where len(days) times daymin=np.zeros(indexmin.shape).astype(np.int) for i,d in enumerate(days): daymin[np.where(indexmin==i)]=d Any better suggestions? Regards Luke Pinner ------ If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. Please consider the environment before printing this email. ------ From gdmcbain at freeshell.org Thu Aug 12 01:34:08 2010 From: gdmcbain at freeshell.org (Geordie McBain) Date: Thu, 12 Aug 2010 15:34:08 +1000 Subject: [Numpy-discussion] Replacing values based on index In-Reply-To: <64102B2FB850C345B04D09C82018F45F0273EDBA@ACT01VEXCCL04.internal.govt> References: <64102B2FB850C345B04D09C82018F45F0273EDBA@ACT01VEXCCL04.internal.govt> Message-ID: 2010/8/12 Pinner, Luke : > Apologies if this has been asked many times before, but I'm having > trouble even coming up with the right terms to search on. > > I have an x,y,date stack of satellite images from which I need to know > on what date the max and min values occur. I can calculate the index at > which the min/max values occur, but how do I join that back to the date? > > A simple example case: > > import numpy as np > > #Date when image was captured, > #images are in the stack in this order > days=[10, 20, 31, 41, 51, 59, 69, 79, 90, 100] > > #dummy array representing something like > #"stackofimages.argmin(axis=2)" > indexmin=np.array([[0, 2, 1], > ? ? ? ? ? ? ? ? ? [6, 7, 3], > ? ? ? ? ? ? ? ? ? [5, 5, 8]]) > > #I want to convert indexmin to "daymin" i.e > #[[10, 31, 20], > # [69, 79, 41], > # [59, 59, 90]] > #Where the each element of indexmin represent the value > #at the corresponding index in the days list/array. > > #What I am currently doing is below, > #which is _rather_ slow on large arrays > #as I'm calling np.where len(days) times > daymin=np.zeros(indexmin.shape).astype(np.int) > for i,d in enumerate(days): > ? ?daymin[np.where(indexmin==i)]=d > > Any better suggestions? Convert days to np.array, then you can index it with your indexmin: days = [10, 20, 31, 41, 51, 59, 69, 79, 90, 100] mydays = np.array (days) indexmin=np.array([[0, 2, 1], [6, 7, 3], [5, 5, 8]]) print mydays[indexmin] => [[10 31 20] [69 79 41] [59 59 90]] From Luke.Pinner at environment.gov.au Thu Aug 12 01:52:43 2010 From: Luke.Pinner at environment.gov.au (Pinner, Luke) Date: Thu, 12 Aug 2010 15:52:43 +1000 Subject: [Numpy-discussion] Replacing values based on index In-Reply-To: References: <64102B2FB850C345B04D09C82018F45F0273EDBA@ACT01VEXCCL04.internal.govt> Message-ID: <64102B2FB850C345B04D09C82018F45F0273EDBB@ACT01VEXCCL04.internal.govt> Simple as that... Thanks! -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Geordie McBain Convert days to np.array, then you can index it with your indexmin: days = [10, 20, 31, 41, 51, 59, 69, 79, 90, 100] mydays = np.array (days) indexmin=np.array([[0, 2, 1], [6, 7, 3], [5, 5, 8]]) print mydays[indexmin] => [[10 31 20] [69 79 41] [59 59 90]] _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ------ If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. Please consider the environment before printing this email. ------ From ralf.gommers at googlemail.com Thu Aug 12 10:35:32 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 12 Aug 2010 22:35:32 +0800 Subject: [Numpy-discussion] 1.5.0 release notes Message-ID: Hi all, Can I ask all of you to please check if there's anything you still have to add to the release notes for 1.5.0 by this Sunday? Also, what is the status of PEP 3118 compatibility? Can someone who knows write a few sentences on that? Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Aug 12 10:52:05 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 12 Aug 2010 22:52:05 +0800 Subject: [Numpy-discussion] Numpy datetime support In-Reply-To: References: Message-ID: On Thu, Aug 12, 2010 at 4:24 AM, John Salvatier wrote: > Hello, > > I am an eager for numpy datetime support, but I have not heard about it > recently. Last I heard it was taken out of 1.4 because of some compatibility > issues. When will datetime be a full numpy feature? > > It will also not be in 1.5 for that same reason. It is still in trunk and will be included in 2.0. The timing of that release depends a bit on when other planned changes that break backward compatibility (notably a refactoring of numpy internals by some Enthought devs) are done. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Aug 12 12:28:19 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 12 Aug 2010 10:28:19 -0600 Subject: [Numpy-discussion] Bug report. Message-ID: Here . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwmsmith at gmail.com Thu Aug 12 14:40:36 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 12 Aug 2010 13:40:36 -0500 Subject: [Numpy-discussion] Bento question: Fortran 90 & Cython support Message-ID: Hi, I'm very interested in Bento and think it will be a killer project. My question: do you anticipate supporting Fortran 90 within Bento, or will that be delegated to an external build tool? There are some intricacies with Fortran 90 that make it difficult to use with the usual configure-then-build ordering, specifically when the configure step depends on .mod files that don't exist until after compilation. Also, what about projects with Pyrex/Cython sources: will Bento internally support .pyx files? Would it be possible to add a command in bento/commands for Fortran 90 / Cython support, or would you recommend doing this through waf/scons? Thanks, Kurt From gberbeglia at gmail.com Thu Aug 12 15:59:17 2010 From: gberbeglia at gmail.com (gerardob) Date: Thu, 12 Aug 2010 12:59:17 -0700 (PDT) Subject: [Numpy-discussion] two dimensional array of sets Message-ID: <29422693.post@talk.nabble.com> Hello, this is a very basic question but i don't know the answer. I would like to construct a two dimensional array A, such that A[i][j] contains a set of numbers associated to the pair (i,j). For example, A[i][j] can be all the numbers that are written as i^n*j*5-n for some all n=1,..,5 Is numpy what i need? if not which is the way to define such arrays. I mean, how do i declare an 2-dimensional array of a given size (for after start filling it with specific sets)? Thanks. -- View this message in context: http://old.nabble.com/two-dimensional-array-of-sets-tp29422693p29422693.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From dwf at cs.toronto.edu Thu Aug 12 17:36:03 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 12 Aug 2010 17:36:03 -0400 Subject: [Numpy-discussion] two dimensional array of sets In-Reply-To: <29422693.post@talk.nabble.com> References: <29422693.post@talk.nabble.com> Message-ID: On 2010-08-12, at 3:59 PM, gerardob wrote: > > Hello, this is a very basic question but i don't know the answer. > > I would like to construct a two dimensional array A, such that A[i][j] > contains a set of numbers associated to the pair (i,j). For example, A[i][j] > can be all the numbers that are written as i^n*j*5-n for some all n=1,..,5 > > Is numpy what i need? if not which is the way to define such arrays. I mean, > how do i declare an 2-dimensional array of a given size (for after start > filling it with specific sets)? The easiest way to do this would be >>> import numpy as np >>> n = 2 >>> i, j = np.ogrid[1:6, 1:6] >>> arr = i**n * j*5 - n >>> print arr array([[ 3, 8, 13, 18, 23], [ 18, 38, 58, 78, 98], [ 43, 88, 133, 178, 223], [ 78, 158, 238, 318, 398], [123, 248, 373, 498, 623]]) David From geometrian at gmail.com Thu Aug 12 17:41:37 2010 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 12 Aug 2010 14:41:37 -0700 Subject: [Numpy-discussion] two dimensional array of sets In-Reply-To: <29422693.post@talk.nabble.com> References: <29422693.post@talk.nabble.com> Message-ID: Hi, You can use np.mgrid to construct a grid of coordinates. From there, you can make your new array based on these coordinates however you like. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From gberbeglia at gmail.com Thu Aug 12 17:54:06 2010 From: gberbeglia at gmail.com (gerardob) Date: Thu, 12 Aug 2010 14:54:06 -0700 (PDT) Subject: [Numpy-discussion] two dimensional array of sets In-Reply-To: References: <29422693.post@talk.nabble.com> Message-ID: <29423645.post@talk.nabble.com> As i wrote, the elements of each A[i][j] are sets and not numbers. David Warde-Farley-2 wrote: > > On 2010-08-12, at 3:59 PM, gerardob wrote: > >> >> Hello, this is a very basic question but i don't know the answer. >> >> I would like to construct a two dimensional array A, such that A[i][j] >> contains a set of numbers associated to the pair (i,j). For example, >> A[i][j] >> can be all the numbers that are written as i^n*j*5-n for some all >> n=1,..,5 >> >> Is numpy what i need? if not which is the way to define such arrays. I >> mean, >> how do i declare an 2-dimensional array of a given size (for after start >> filling it with specific sets)? > > The easiest way to do this would be > >>>> import numpy as np >>>> n = 2 >>>> i, j = np.ogrid[1:6, 1:6] >>>> arr = i**n * j*5 - n >>>> print arr > array([[ 3, 8, 13, 18, 23], > [ 18, 38, 58, 78, 98], > [ 43, 88, 133, 178, 223], > [ 78, 158, 238, 318, 398], > [123, 248, 373, 498, 623]]) > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- View this message in context: http://old.nabble.com/two-dimensional-array-of-sets-tp29422693p29423645.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From ben.root at ou.edu Thu Aug 12 19:05:19 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 12 Aug 2010 18:05:19 -0500 Subject: [Numpy-discussion] two dimensional array of sets In-Reply-To: <29422693.post@talk.nabble.com> References: <29422693.post@talk.nabble.com> Message-ID: On Thu, Aug 12, 2010 at 2:59 PM, gerardob wrote: > > Hello, this is a very basic question but i don't know the answer. > > I would like to construct a two dimensional array A, such that A[i][j] > contains a set of numbers associated to the pair (i,j). For example, > A[i][j] > can be all the numbers that are written as i^n*j*5-n for some all n=1,..,5 > > Is numpy what i need? if not which is the way to define such arrays. I > mean, > how do i declare an 2-dimensional array of a given size (for after start > filling it with specific sets)? > > Thanks. > > > Because you want to hold a set, it might possibly be better to use python lists and list comprehensions. However, you can use numpy to help you out... import numpy as np rows = 5 cols = 5 n = np.arange(1, 6) a = [[set((i**n) * (j*(5 - n))) for j in range(1, cols+1)] for i in range(1,rows+1)] The first row is a[0] (which is i == 1) : [set([0, 1, 2, 3, 4]), set([8, 0, 2, 4, 6]), set([0, 9, 3, 12, 6]), set([16, 8, 12, 4, 0]), set([0, 10, 20, 5, 15])] I hope this helps, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Aug 12 14:37:22 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 12 Aug 2010 19:37:22 +0100 Subject: [Numpy-discussion] 1.5.0 release notes In-Reply-To: References: Message-ID: <1281638242.8059.2.camel@Nokia-N900-42-11> > Also, what is the status of PEP 3118 compatibility? Can someone who knows > write a few sentences on that? The features in the spec are (should be) completely supported now. You can probably add the following: The new buffer protocol described by PEP 3118 is fully supported in this version of Numpy. On Python versions >= 2.6 Numpy arrays expose the buffer interface, and array(), asarray() and other functions accept new-style buffers as input. -- Pauli Virtanen From dwf at cs.toronto.edu Thu Aug 12 19:51:26 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 12 Aug 2010 19:51:26 -0400 Subject: [Numpy-discussion] Broken links on new.scipy In-Reply-To: References: Message-ID: <7F28CA8B-34E7-4E44-9F26-3921974B949A@cs.toronto.edu> Thanks Gokhan, I'll push that tonight (along with lots of other changes). I just got the necessary permissions the other day and haven't had much time this week. David On 2010-08-06, at 3:30 PM, G?khan Sever wrote: > Hi, > > @ http://new.scipy.org/download.html numpy and scipy links for Fedora is broken. > > Could you update the links with these? > > https://admin.fedoraproject.org/pkgdb/acls/name/numpy > > https://admin.fedoraproject.org/pkgdb/acls/name/scipy > > Thanks. > > -- > G?khan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Aug 12 20:18:55 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 12 Aug 2010 20:18:55 -0400 Subject: [Numpy-discussion] two dimensional array of sets In-Reply-To: <29423645.post@talk.nabble.com> References: <29422693.post@talk.nabble.com> <29423645.post@talk.nabble.com> Message-ID: <26AF0C32-F40F-4250-A7E9-CBC6E499CB66@cs.toronto.edu> On 2010-08-12, at 5:54 PM, gerardob wrote: > As i wrote, the elements of each A[i][j] are sets and not numbers. Ah, missed that part. Assuming you don't need/want these sets to be mutable or differently sized, you could do something like this. Here we make 'n' an array of the numbers 0 to 49, but you could do other things. We immediately reshape it to a 3D array with the 50 elements being the "depth" dimension. >>>> n = np.arange(50).reshape(1, 1, 50) >>>> i, j, _ = np.ogrid[1:6, 1:6, 0:1] # adding a dummy dimension Look in the NumPy docs about "broadcasting" to see why this works the way it does, also have a gander at the contents of the i and j arrays and their .shape properties. >>>> arr = i**n * j*5 - n >>>> print arr[0, 0] [ 5 4 3 2 1 0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44] If you want the sets to be proper Python sets then you might want to look at Ben's reply, but you can accomplish many operations on sets using 1-d NumPy arrays. Have a look at help(np.lib.arraysetops) for a list of functions. Also, NumPy can only work with fixed-size data types so if you need to work with numbers bigger than 2^63 - 1 (or smaller than -2^63) than you ought to use Ben's method. David David From lutz.maibaum at gmail.com Thu Aug 12 21:26:34 2010 From: lutz.maibaum at gmail.com (Lutz Maibaum) Date: Thu, 12 Aug 2010 18:26:34 -0700 Subject: [Numpy-discussion] ndarray.copy does not accept keyword argument Message-ID: According to the docstring, ndarray.copy should accept a keyword argument "order". This doesn't seem to work for me: >>> np.array([[1,2],[3,4]]).copy(order='C') Traceback (most recent call last): File "", line 1, in TypeError: copy() takes no keyword arguments Calling ndarray.copy with a positional argument seems to work, but this behavior seems inconsistent with the documentation and other occurrences of the "order" keyword. Is this a bug, or am I misunderstanding the documentation? Thanks, Lutz From david at silveregg.co.jp Thu Aug 12 21:51:46 2010 From: david at silveregg.co.jp (David) Date: Fri, 13 Aug 2010 10:51:46 +0900 Subject: [Numpy-discussion] Bento question: Fortran 90 & Cython support In-Reply-To: References: Message-ID: <4C64A532.8070009@silveregg.co.jp> Hi Kurt, On 08/13/2010 03:40 AM, Kurt Smith wrote: > Hi, > > I'm very interested in Bento and think it will be a killer project. > > My question: do you anticipate supporting Fortran 90 within Bento, or > will that be delegated to an external build tool? Bento does not support building at all. The idea is that you will be able to plug your build tool of choice if desired, and bento will just gives you a very high level description of your extensions. Right now, I am working on yaku, which is a simple build tool designed as a library, and would be used by bento by default. If I can build scipy with it and keep it simple, I will integrate yaku and bento. This means at least basic fortran spuport. Otherwise, I may just give up yaku - if it ends up being as big as say waf, there is really no point in it. I was not using waf directly because previous to 1.6, using waf as a library was not as easy as expected (much better than scons, though), but this is changing in 1.6. This means I will also need to add fortran support to waf. I really like waf but was a bit concerned with the lack of usage - now that samba is working on it, the prospect of long term support for waf look much better ( http://wiki.samba.org/index.php/Waf). > There are some > intricacies with Fortran 90 that make it difficult to use with the > usual configure-then-build ordering, specifically when the configure > step depends on .mod files that don't exist until after compilation. Could you expand on this ? Nothing prevents the configure step from building some mod files necessary for configuration a priori. > > Also, what about projects with Pyrex/Cython sources: will Bento > internally support .pyx files? You can already build extensions with cython with bento+yaku. You just add .pyx files as sources in the Extension section of the bento.info, and it is then up to the build tool to deal with .pyx. You can see an example in the port of nipy build to bento, which has cython files: http://github.com/cournape/nipy/tree/bento_build Note that even though the bento.info format itself is pretty stable, anything that goes in bscript files (to customize build) keeps changing and is highly unstable. I still have no clear idea about the API (scipy's build idiosyncraties keep breaking the assumptions I have made so far :) ). To somewhat deal with the unstability API-wise, you can include a copy of bento+yaku in your project, as I have done in nipy. It is a self-contained file which is about 350 kb (and down to 80 kb if you don't care about supporting building windows installers). cheers, David From kwmsmith at gmail.com Fri Aug 13 09:49:13 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 13 Aug 2010 08:49:13 -0500 Subject: [Numpy-discussion] ndarray.copy does not accept keyword argument In-Reply-To: References: Message-ID: On Thu, Aug 12, 2010 at 8:26 PM, Lutz Maibaum wrote: > According to the docstring, ndarray.copy should accept a keyword argument "order". This doesn't seem to work for me: > >>>> np.array([[1,2],[3,4]]).copy(order='C') > Traceback (most recent call last): > ?File "", line 1, in > TypeError: copy() takes no keyword arguments > > Calling ndarray.copy with a positional argument seems to work, but this behavior seems inconsistent with the documentation and other occurrences of the "order" keyword. Is this a bug, or am I misunderstanding the documentation? http://projects.scipy.org/numpy/ticket/1581 Kurt > > Thanks, > > ?Lutz > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwmsmith at gmail.com Fri Aug 13 10:28:12 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 13 Aug 2010 09:28:12 -0500 Subject: [Numpy-discussion] Bento question: Fortran 90 & Cython support In-Reply-To: <4C64A532.8070009@silveregg.co.jp> References: <4C64A532.8070009@silveregg.co.jp> Message-ID: On Thu, Aug 12, 2010 at 8:51 PM, David wrote: > Hi Kurt, > > On 08/13/2010 03:40 AM, Kurt Smith wrote: >> Hi, >> >> I'm very interested in Bento and think it will be a killer project. >> >> My question: do you anticipate supporting Fortran 90 within Bento, or >> will that be delegated to an external build tool? > > Bento does not support building at all. The idea is that you will be > able to plug your build tool of choice if desired, and bento will just > gives you a very high level description of your extensions. > > Right now, I am working on yaku, which is a simple build tool designed > as a library, and would be used by bento by default. If I can build > scipy with it and keep it simple, I will integrate yaku and bento. This > means at least basic fortran spuport. Otherwise, I may just give up yaku > - if it ends up being as big as say waf, there is really no point in it. > > I was not using waf directly because previous to 1.6, using waf as a > library was not as easy as expected (much better than scons, though), > but this is changing in 1.6. This means I will also need to add fortran > support to waf. I really like waf but was a bit concerned with the lack > of usage - now that samba is working on it, the prospect of long term > support for waf look much better ( > http://wiki.samba.org/index.php/Waf). > This is all very good to hear. I've been struggling with numpy.distutils for fwrap, and it is a nightmare. My current problem---requiring yet-another-monkeypatch---is getting try_compile in the 'config' stage to work with Fortran 90 sources and .mod files that have to be generated before the 'config' stage begins. Sounds like the best long-term solution is to put effort into better fortran support in waf, unless you plan on yaku's fortran support covering all cases (fortran 77/9X, .mod files, etc.). I'll start looking into that. Is there a repo somewhere, or are you contributing to waf directly? >> ?There are some >> intricacies with Fortran 90 that make it difficult to use with the >> usual configure-then-build ordering, specifically when the configure >> step depends on .mod files that don't exist until after compilation. > > Could you expand on this ? Nothing prevents the configure step from > building some mod files necessary for configuration a priori. > Are you referring to waf? Or are you speaking of Bento/yaku? I don't know waf that well, but my impression was that the configure & build stages are fairly separate with compilation only in the latter. But that was just an initial impression from a quick look at the source. A somewhat separate question: Since both waf & bento have 'configure' steps how do they coordinate with each other, i.e. what is the 'responsibility' for each configure step? (same goes for bento/scons, or any other build tool, I reckon.) >> >> Also, what about projects with Pyrex/Cython sources: will Bento >> internally support .pyx files? > > You can already build extensions with cython with bento+yaku. You just > add .pyx files as sources in the Extension section of the bento.info, > and it is then up to the build tool to deal with .pyx. > > You can see an example in the port of nipy build to bento, which has > cython files: > > http://github.com/cournape/nipy/tree/bento_build > > Note that even though the bento.info format itself is pretty stable, > anything that goes in bscript files (to customize build) keeps changing > and is highly unstable. I still have no clear idea about the API > (scipy's build idiosyncraties keep breaking the assumptions I have made > so far :) ). > > To somewhat deal with the unstability API-wise, you can include a copy > of bento+yaku in your project, as I have done in nipy. It is a > self-contained file which is about 350 kb (and down to 80 kb if you > don't care about supporting building windows installers). > Interesting. Kinda like waf's recommendation to include waf with the project, which I like. Is this recommended for bento/yaku, too, or just a current stopgap? > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Fri Aug 13 11:20:34 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 14 Aug 2010 00:20:34 +0900 Subject: [Numpy-discussion] Bento question: Fortran 90 & Cython support In-Reply-To: References: <4C64A532.8070009@silveregg.co.jp> Message-ID: On Fri, Aug 13, 2010 at 11:28 PM, Kurt Smith wrote: > On Thu, Aug 12, 2010 at 8:51 PM, David wrote: >> Hi Kurt, >> >> On 08/13/2010 03:40 AM, Kurt Smith wrote: >>> Hi, >>> >>> I'm very interested in Bento and think it will be a killer project. >>> >>> My question: do you anticipate supporting Fortran 90 within Bento, or >>> will that be delegated to an external build tool? >> >> Bento does not support building at all. The idea is that you will be >> able to plug your build tool of choice if desired, and bento will just >> gives you a very high level description of your extensions. >> >> Right now, I am working on yaku, which is a simple build tool designed >> as a library, and would be used by bento by default. If I can build >> scipy with it and keep it simple, I will integrate yaku and bento. This >> means at least basic fortran spuport. Otherwise, I may just give up yaku >> - if it ends up being as big as say waf, there is really no point in it. >> >> I was not using waf directly because previous to 1.6, using waf as a >> library was not as easy as expected (much better than scons, though), >> but this is changing in 1.6. This means I will also need to add fortran >> support to waf. I really like waf but was a bit concerned with the lack >> of usage - now that samba is working on it, the prospect of long term >> support for waf look much better ( >> http://wiki.samba.org/index.php/Waf). >> > > This is all very good to hear. ?I've been struggling with > numpy.distutils for fwrap, and it is a nightmare. ?My current > problem---requiring yet-another-monkeypatch---is getting try_compile > in the 'config' stage to work with Fortran 90 sources and .mod files > that have to be generated before the 'config' stage begins. I am still not clear on the exact issue you are trying to solve: could you just say which file output are you expecting in which order ? > > Sounds like the best long-term solution is to put effort into better > fortran support in waf, unless you plan on yaku's fortran support > covering all cases (fortran 77/9X, .mod files, etc.). I want yaku to be able to build scipy so that I know that it will cover 99 % of python needs if integrated to bento. But I still intend to use waf+bento as the longer term. Waf is just very powerful, and the new codebase at its core is around 3500 lines of code. The only advantage of yaku is to be built from the ground up as a library. > ?I'll start > looking into that. ?Is there a repo somewhere, or are you contributing > to waf directly? There used to be some fortran support in the waf trunk, but this has disappeared with the new trunk for 1.6. I hope to be able to port it this WE. > Are you referring to waf? ?Or are you speaking of Bento/yaku? In general. The configure stage does involve compilation, so I don't really understand your problem. You could do everything you want in the configure stage - the main point is that configure and build are separated, and you don't expect configure stage to change much during development. So everything that is potentially expensive and stable, you put in the configure stage. > > A somewhat separate question: Since both waf & bento have 'configure' > steps how do they coordinate with each other, i.e. what is the > 'responsibility' for each configure step? (same goes for bento/scons, > or any other build tool, I reckon.) You can set up a hook file in the bento.info file, which is a python file containing hooks for each stage (configure, build, etc..). Right now, if you build with yaku, the configure and build hooks give you yaku configure and build contexts, from which you can do whatever you could using yaku directly. In practice, it is a bit more complicated, because I want most of the code to be declarative in principle, with bento driving the process and calling your functions, but sometimes imperative is needed. So right now the whole thing is very messy, especially yaku, and will remain so until I figure it all out :) > Interesting. ?Kinda like waf's recommendation to include waf with the > project, which I like. ?Is this recommended for bento/yaku, too, or > just a current stopgap? It is certainly recommended for now, but I see no reason to disable that feature later. But I also want to make it natural to install bento, so that it can be included in distributions. I hope that once it stabilizes, bento won't need to change much. One of the main rationale to start this is to make it extensible after all, cheers, David From d.l.goldsmith at gmail.com Fri Aug 13 12:47:26 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 13 Aug 2010 09:47:26 -0700 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of In-Reply-To: References: Message-ID: 2010/7/30 St?fan van der Walt > Hi David > > Best of luck with your new position! I hope they don't make you program too > much MATLAB! > After several years now of writing Python and now having written my first on-the-job 15 operational MATLAB LOC, all of which are string, cell array, and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( DG > I appreciate your eye for detail in the docs project, and I hope we won't > lose that. > > Kind regards > St?fan > > On 29 Jul 2010 07:10, "David Goldsmith" wrote: > > July 31 is my last day as paid NumPy/SciPy docs editor/marathon > coordinator; I'm moving on to a programming data analyst position for the Marine > Waters Data Management System Project of the Environmental Assessment > Program in the Washington State Department of Ecology; unfortunately, they > need me to program in MATLAB, but I plan on continuing to use NumPy, etc. > for my fractals, etc. I also intend to continue to contribute to this > community as a volunteer as my time permits, but you probably will be seeing > less of me on the lists. Also, I won't be hosting a Marathon Skypecon this > Friday (but of course others are free to organize one if they want to.) > > Thanks for all your help and support, > > David Goldsmith > Olympia, WA > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -- Privacy is overrated; Identity isn't. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdh2358 at gmail.com Fri Aug 13 12:59:03 2010 From: jdh2358 at gmail.com (John Hunter) Date: Fri, 13 Aug 2010 11:59:03 -0500 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 11:47 AM, David Goldsmith wrote: > 2010/7/30 St?fan van der Walt >> >> Hi David >> >> Best of luck with your new position! I hope they don't make you program >> too much MATLAB! > > After several years now of writing Python and now having written my first > on-the-job 15 operational MATLAB LOC, all of which are string, cell array, > and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( Ahh, cell arrays, they bring back memories. Makes you pine for a dictionary, no? JDH From josef.pktd at gmail.com Fri Aug 13 13:06:23 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 Aug 2010 13:06:23 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 12:59 PM, John Hunter wrote: > On Fri, Aug 13, 2010 at 11:47 AM, David Goldsmith > wrote: >> 2010/7/30 St?fan van der Walt >>> >>> Hi David >>> >>> Best of luck with your new position! I hope they don't make you program >>> too much MATLAB! >> >> After several years now of writing Python and now having written my first >> on-the-job 15 operational MATLAB LOC, all of which are string, cell array, >> and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( > > Ahh, cell arrays, they bring back memories. ?Makes you pine for a > dictionary, no? but there is the matlab fileexchange where you find a recipe for almost anything, which makes being tortured a bit less severe. Josef > > JDH > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From d.l.goldsmith at gmail.com Fri Aug 13 16:04:39 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 13 Aug 2010 13:04:39 -0700 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) Message-ID: > > After several years now of writing Python and now having written my first > > on-the-job 15 operational MATLAB LOC, all of which are string, cell > array, > > and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( > > Ahh, cell arrays, they bring back memories. Makes you pine for a > dictionary, no? > > JDH Not to mention writeline, readline, string concatenation using +, English wording of loops, list comprehension, etc., etc., etc. - if people only knew... DG > -- > Privacy is overrated; Identity isn't. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.holbrook at gmail.com Fri Aug 13 16:27:19 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Fri, 13 Aug 2010 12:27:19 -0800 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: Ugh! I miss all that stuff anytime I have to use matlab. Working with strings sucks especially hard. In fact, I've tried reimplementing half of the tools I wish I had. If anyone cares: http://github.com/jesusabdullah/methlabs I'd love feedback. Perhaps ironically, I haven't used it much, due to not wanting to give my matlabs dependencies. :/ --Josh On Fri, Aug 13, 2010 at 12:04 PM, David Goldsmith wrote: > >> > After several years now of writing Python and now having written my >> > first >> > on-the-job 15 operational MATLAB LOC, all of which are string, cell >> > array, >> > and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( >> >> Ahh, cell arrays, they bring back memories. ?Makes you pine for a >> dictionary, no? >> >> JDH > > Not to mention writeline, readline, string concatenation using +, English > wording of loops, list comprehension, etc., etc., etc. - if people only > knew... > > DG > >> >> -- > > Privacy is overrated; Identity isn't. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ben.root at ou.edu Fri Aug 13 16:41:08 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 13 Aug 2010 15:41:08 -0500 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: @Josh: Awesome name. Very fitting... Another thing that I really love about matplotlib that drove me nuts in Matlab was being unable to use multiple colormaps in the same figure. And don't get me started with the map plotting tools and pcolor! Ben Root On Fri, Aug 13, 2010 at 3:27 PM, Joshua Holbrook wrote: > Ugh! I miss all that stuff anytime I have to use matlab. Working with > strings sucks especially hard. > > In fact, I've tried reimplementing half of the tools I wish I had. If > anyone cares: > > http://github.com/jesusabdullah/methlabs > > I'd love feedback. Perhaps ironically, I haven't used it much, due to > not wanting to give my matlabs dependencies. :/ > > --Josh > > > > On Fri, Aug 13, 2010 at 12:04 PM, David Goldsmith > wrote: > > > >> > After several years now of writing Python and now having written my > >> > first > >> > on-the-job 15 operational MATLAB LOC, all of which are string, cell > >> > array, > >> > and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( > >> > >> Ahh, cell arrays, they bring back memories. Makes you pine for a > >> dictionary, no? > >> > >> JDH > > > > Not to mention writeline, readline, string concatenation using +, English > > wording of loops, list comprehension, etc., etc., etc. - if people only > > knew... > > > > DG > > > >> > >> -- > > > > Privacy is overrated; Identity isn't. > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdh2358 at gmail.com Fri Aug 13 16:59:23 2010 From: jdh2358 at gmail.com (John Hunter) Date: Fri, 13 Aug 2010 15:59:23 -0500 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 3:41 PM, Benjamin Root wrote: > @Josh: Awesome name.? Very fitting... > > Another thing that I really love about matplotlib that drove me nuts in > Matlab was being unable to use multiple colormaps in the same figure. Funny -- this was one of the *first* things I thought about when writing mpl. That limitation drove me nuts too. And while we're dissing matlab, the one-function-per-file development process that matlab seems to inspire (I did it too) is hard to fathom in retrospect. http://github.com/jesusabdullah/methlabs/blob/master/modules/extra/elemsof.m JDH From josh.holbrook at gmail.com Fri Aug 13 17:09:26 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Fri, 13 Aug 2010 13:09:26 -0800 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: Yeah, I don't really like it either. Is there a good way to put multiple global functions in a file without matlab getting upset? Cause honestly, If I could dump everything in one file and be able to use it everywhere. --Josh On Fri, Aug 13, 2010 at 12:59 PM, John Hunter wrote: > On Fri, Aug 13, 2010 at 3:41 PM, Benjamin Root wrote: >> @Josh: Awesome name.? Very fitting... >> >> Another thing that I really love about matplotlib that drove me nuts in >> Matlab was being unable to use multiple colormaps in the same figure. > > Funny -- this was one of the *first* things I thought about when > writing mpl. ?That limitation drove me nuts too. > > And while we're dissing matlab, the one-function-per-file development > process that matlab seems to inspire (I did it too) is hard to fathom > in retrospect. > > http://github.com/jesusabdullah/methlabs/blob/master/modules/extra/elemsof.m > > JDH > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josh.holbrook at gmail.com Fri Aug 13 17:11:21 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Fri, 13 Aug 2010 13:11:21 -0800 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: > Cause honestly, If I could dump everything in one file and be able to > use it everywhere. ...I could. Sorry. --Josh On Fri, Aug 13, 2010 at 1:09 PM, Joshua Holbrook wrote: > Yeah, I don't really like it either. Is there a good way to put > multiple global functions in a file without matlab getting upset? > Cause honestly, If I could dump everything in one file and be able to > use it everywhere. > > --Josh > > On Fri, Aug 13, 2010 at 12:59 PM, John Hunter wrote: >> On Fri, Aug 13, 2010 at 3:41 PM, Benjamin Root wrote: >>> @Josh: Awesome name.? Very fitting... >>> >>> Another thing that I really love about matplotlib that drove me nuts in >>> Matlab was being unable to use multiple colormaps in the same figure. >> >> Funny -- this was one of the *first* things I thought about when >> writing mpl. ?That limitation drove me nuts too. >> >> And while we're dissing matlab, the one-function-per-file development >> process that matlab seems to inspire (I did it too) is hard to fathom >> in retrospect. >> >> http://github.com/jesusabdullah/methlabs/blob/master/modules/extra/elemsof.m >> >> JDH >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From josef.pktd at gmail.com Fri Aug 13 19:48:22 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 Aug 2010 19:48:22 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 5:11 PM, Joshua Holbrook wrote: >> Cause honestly, If I could dump everything in one file and be able to >> use it everywhere. > > ...I could. > > Sorry. The main thing I was missing in matlab are namespaces, everything the equivalent of form numpy import * form pylab import * from scipy import * from all_my_other_packages import * So, for a while I put many functions in one file and imported them as function handles. I liked that I could edit a plot interactively and save as script. This saved me a lot of time reading and figuring out the docs. The new style classes in matlab look almost like python, but I haven't worked with them yet. Josef > > --Josh > > On Fri, Aug 13, 2010 at 1:09 PM, Joshua Holbrook > wrote: >> Yeah, I don't really like it either. Is there a good way to put >> multiple global functions in a file without matlab getting upset? >> Cause honestly, If I could dump everything in one file and be able to >> use it everywhere. >> >> --Josh >> >> On Fri, Aug 13, 2010 at 12:59 PM, John Hunter wrote: >>> On Fri, Aug 13, 2010 at 3:41 PM, Benjamin Root wrote: >>>> @Josh: Awesome name.? Very fitting... >>>> >>>> Another thing that I really love about matplotlib that drove me nuts in >>>> Matlab was being unable to use multiple colormaps in the same figure. >>> >>> Funny -- this was one of the *first* things I thought about when >>> writing mpl. ?That limitation drove me nuts too. >>> >>> And while we're dissing matlab, the one-function-per-file development >>> process that matlab seems to inspire (I did it too) is hard to fathom >>> in retrospect. >>> >>> http://github.com/jesusabdullah/methlabs/blob/master/modules/extra/elemsof.m >>> >>> JDH >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Sat Aug 14 03:01:46 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Aug 2010 15:01:46 +0800 Subject: [Numpy-discussion] 3K: fromfile/tofile errors on Windows Message-ID: ====================================================================== ERROR: test_multiarray.TestIO.test_ascii ---------------------------------------------------------------------- Traceback (most recent call last): File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest self.test(*self.arg) File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1017, in test_ascii self._check_from('1 , 2 , 3 , 4', [1.,2.,3.,4.], sep=',') File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 980, in _check_from y = np.fromfile(self.filename, **kw) IOError: first argument must be an open file ====================================================================== ERROR: test_multiarray.TestIO.test_ascii ---------------------------------------------------------------------- Traceback (most recent call last): File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 370, in tearDown try_run(self.inst, ('teardown', 'tearDown')) File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\util.py", line 469, in try_run return func() File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 918, in tearDown os.unlink(self.filename) WindowsError: [Error 32] Sharing violation: 'z:\\var\\folders\\uu\\uuxfo1nlfae4yyypscz-xe+++ti\\-tmp-\\tmpdtktwh' And 18 more like that. These are the only errors with 1.5.x and Python 3.1.2 on Windows XP. Ticket: http://projects.scipy.org/numpy/ticket/1583. Anyone have an idea? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Aug 14 03:32:44 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Aug 2010 15:32:44 +0800 Subject: [Numpy-discussion] review request Message-ID: Hi, I'd appreciate a review of these changes: http://github.com/rgommers/numpy/tree/changes-for-1.5 http://github.com/rgommers/numpy/tree/deprecations Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Aug 14 06:29:38 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Aug 2010 18:29:38 +0800 Subject: [Numpy-discussion] cython/swigpyrex examples and docs In-Reply-To: References: Message-ID: On Sat, Jul 31, 2010 at 2:32 PM, Ralf Gommers wrote: > > > On Sat, Jul 24, 2010 at 6:38 PM, Pauli Virtanen wrote: > >> Sat, 24 Jul 2010 18:22:33 +0800, Ralf Gommers wrote: >> > There is a bug report asking for the example files under doc/cython, >> > doc/swig etc. to be included in the released binaries. This thread, >> > http://thread.gmane.org/gmane.comp.python.numeric.general/26913/ >> focus=26917 >> , >> > seems to indicate the docs should still be integrated in the main >> > documentation. What do we want to do with the example files, are they >> > still up to date? >> >> There are two points here, I guess: >> >> - the files should be integrated to the docs >> >> The one file to be integrated is numpy_swig.txt. How about putting it into > the reference guide as a separate chapter titled "Numpy and SWIG", right > after "Numpy internals"? It's already in reST format so it's not much work. > > The swig/testing.txt file should probably not be included. Cython and Pyrex > dirs do not have docs. > Actually, testing.txt does look useful for the reference guide too, so I included it as well. This addresses both your points I think, please have a look: http://github.com/rgommers/numpy/commits/examples-swig-cython Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Aug 14 08:43:01 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 Aug 2010 06:43:01 -0600 Subject: [Numpy-discussion] review request In-Reply-To: References: Message-ID: On Sat, Aug 14, 2010 at 1:32 AM, Ralf Gommers wrote: > Hi, > > I'd appreciate a review of these changes: > http://github.com/rgommers/numpy/tree/changes-for-1.5 > http://github.com/rgommers/numpy/tree/deprecations > > Looks good to me as far as I am familiar with things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Aug 14 09:11:54 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Aug 2010 21:11:54 +0800 Subject: [Numpy-discussion] py3k execfile('setup.py') In-Reply-To: References: Message-ID: On Fri, Jul 30, 2010 at 12:04 AM, Pauli Virtanen wrote: > Thu, 29 Jul 2010 23:39:19 +0800, Ralf Gommers wrote: > > The execfile builtin has disappeared in python 3.x, so I'm trying to > > find another solution for the use of it in setupegg.py. So far I've > > tried > > I'd do something like this in "setup.py": > > ... > > + if os.environ.get('USE_SETUPTOOLS'): > + import setuptools > from numpy.distutils.core import setup > ... > > And then 'setupegg.py' is trivial to write. > > But no idea why it hangs, though. You could maybe also try 'import setup' > Actually, it was hanging because of bdist_mpkg which I was using. Now your suggestion above and using imp both can give me an egg install. "import setup" doesn't work, numpy.distutils gets unhappy somewhere. Installing into a local virtualenv (with either of the methods that work) looks good. Into /Library/Frameworks/..., nose refuses to run any tests, due to the files being executable. But I think that's normal. Changing permissions to 644 manually it all looks good again. I prefer to use imp because it doesn't change setup.py. Adding USE_SETUPTOOLS to setup.py could give the impression that it may actually be a good idea to use this, which it's not. Proposed fix: http://github.com/rgommers/numpy/tree/setupegg What's left is to create an OS X installer. bdist_mpkg is not maintained anymore, so I'm not sure what the best option is. Any suggestions? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Aug 14 09:36:01 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Aug 2010 21:36:01 +0800 Subject: [Numpy-discussion] review request In-Reply-To: References: Message-ID: On Sat, Aug 14, 2010 at 8:43 PM, Charles R Harris wrote: > > > On Sat, Aug 14, 2010 at 1:32 AM, Ralf Gommers > wrote: > >> Hi, >> >> I'd appreciate a review of these changes: >> http://github.com/rgommers/numpy/tree/changes-for-1.5 >> http://github.com/rgommers/numpy/tree/deprecations >> >> > Looks good to me as far as I am familiar with things. > > Thanks. In it goes then. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.holbrook at gmail.com Sat Aug 14 16:02:52 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Sat, 14 Aug 2010 12:02:52 -0800 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 3:48 PM, wrote: > On Fri, Aug 13, 2010 at 5:11 PM, Joshua Holbrook > wrote: >>> Cause honestly, If I could dump everything in one file and be able to >>> use it everywhere. >> >> ...I could. >> >> Sorry. > > The main thing I was missing in matlab are namespaces, everything the > equivalent of > form numpy import * > form pylab import * > from scipy import * > from all_my_other_packages import * > > So, for a while I put many functions in one file and imported them as > function handles. > > I liked that I could edit a plot interactively and save as script. > This saved me a lot of time reading and figuring out the docs. > The new style classes in matlab look almost like python, but I haven't > worked with them yet. > > Josef > > >> >> --Josh >> >> On Fri, Aug 13, 2010 at 1:09 PM, Joshua Holbrook >> wrote: >>> Yeah, I don't really like it either. Is there a good way to put >>> multiple global functions in a file without matlab getting upset? >>> Cause honestly, If I could dump everything in one file and be able to >>> use it everywhere. >>> >>> --Josh >>> >>> On Fri, Aug 13, 2010 at 12:59 PM, John Hunter wrote: >>>> On Fri, Aug 13, 2010 at 3:41 PM, Benjamin Root wrote: >>>>> @Josh: Awesome name.? Very fitting... >>>>> >>>>> Another thing that I really love about matplotlib that drove me nuts in >>>>> Matlab was being unable to use multiple colormaps in the same figure. >>>> >>>> Funny -- this was one of the *first* things I thought about when >>>> writing mpl. ?That limitation drove me nuts too. >>>> >>>> And while we're dissing matlab, the one-function-per-file development >>>> process that matlab seems to inspire (I did it too) is hard to fathom >>>> in retrospect. >>>> >>>> http://github.com/jesusabdullah/methlabs/blob/master/modules/extra/elemsof.m >>>> >>>> JDH >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > So, for a while I put many functions in one file and imported them as function handles. Ooh! How do you do that? --Josh From josef.pktd at gmail.com Sat Aug 14 16:47:15 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 14 Aug 2010 16:47:15 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: On Sat, Aug 14, 2010 at 4:02 PM, Joshua Holbrook wrote: > On Fri, Aug 13, 2010 at 3:48 PM, ? wrote: >> On Fri, Aug 13, 2010 at 5:11 PM, Joshua Holbrook >> wrote: >>>> Cause honestly, If I could dump everything in one file and be able to >>>> use it everywhere. >>> >>> ...I could. >>> >>> Sorry. >> >> The main thing I was missing in matlab are namespaces, everything the >> equivalent of >> form numpy import * >> form pylab import * >> from scipy import * >> from all_my_other_packages import * >> >> So, for a while I put many functions in one file and imported them as >> function handles. >> >> I liked that I could edit a plot interactively and save as script. >> This saved me a lot of time reading and figuring out the docs. >> The new style classes in matlab look almost like python, but I haven't >> worked with them yet. >> >> Josef >> >> >>> >>> --Josh >>> >>> On Fri, Aug 13, 2010 at 1:09 PM, Joshua Holbrook >>> wrote: >>>> Yeah, I don't really like it either. Is there a good way to put >>>> multiple global functions in a file without matlab getting upset? >>>> Cause honestly, If I could dump everything in one file and be able to >>>> use it everywhere. >>>> >>>> --Josh >>>> >>>> On Fri, Aug 13, 2010 at 12:59 PM, John Hunter wrote: >>>>> On Fri, Aug 13, 2010 at 3:41 PM, Benjamin Root wrote: >>>>>> @Josh: Awesome name.? Very fitting... >>>>>> >>>>>> Another thing that I really love about matplotlib that drove me nuts in >>>>>> Matlab was being unable to use multiple colormaps in the same figure. >>>>> >>>>> Funny -- this was one of the *first* things I thought about when >>>>> writing mpl. ?That limitation drove me nuts too. >>>>> >>>>> And while we're dissing matlab, the one-function-per-file development >>>>> process that matlab seems to inspire (I did it too) is hard to fathom >>>>> in retrospect. >>>>> >>>>> http://github.com/jesusabdullah/methlabs/blob/master/modules/extra/elemsof.m >>>>> >>>>> JDH >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> So, for a while I put many functions in one file and imported them as > function handles. > > Ooh! How do you do that? something like, api is a struct that has function handles and variables attached function api = setAPI api.fcn1 = @fcn1; api.fcn2 = @fcn2; api.showvar1 = @showvar1; api.getvar = @getvar; api.setglobalVar = @setglobalVar; api.getglobalVar = @getglobalVar; %api.globalVar = globalVar; end %setAPI for the fun of it: I attach one of my first matlab files to check out how the scope of nested functions works, i.e. the workspace that is attached to functions and nested functions. It's not very useful except for learning matlab 2006. I don't find an example for sub functions instead of nested functions right now. And I soon gave up trying to program matlab like python. Josef (as off-topic as most of the second part of this thread) > > --Josh > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: getnestedstruct.m Type: application/x-matlab Size: 2638 bytes Desc: not available URL: From gary.pajer at gmail.com Sat Aug 14 22:34:19 2010 From: gary.pajer at gmail.com (Gary Pajer) Date: Sat, 14 Aug 2010 22:34:19 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: Similar situation here. Been Matlab-free for about ten years, but my new employer ... I started looking at the code, and my stomach sank when I first read if (nargin = 1) Oh, it hurts!! On Fri, Aug 13, 2010 at 4:41 PM, Benjamin Root wrote: > @Josh: Awesome name. Very fitting... > > Another thing that I really love about matplotlib that drove me nuts in > Matlab was being unable to use multiple colormaps in the same figure. > > And don't get me started with the map plotting tools and pcolor! > > Ben Root > > > On Fri, Aug 13, 2010 at 3:27 PM, Joshua Holbrook wrote: > >> Ugh! I miss all that stuff anytime I have to use matlab. Working with >> strings sucks especially hard. >> >> In fact, I've tried reimplementing half of the tools I wish I had. If >> anyone cares: >> >> http://github.com/jesusabdullah/methlabs >> >> I'd love feedback. Perhaps ironically, I haven't used it much, due to >> not wanting to give my matlabs dependencies. :/ >> >> --Josh >> >> >> >> On Fri, Aug 13, 2010 at 12:04 PM, David Goldsmith >> wrote: >> > >> >> > After several years now of writing Python and now having written my >> >> > first >> >> > on-the-job 15 operational MATLAB LOC, all of which are string, cell >> >> > array, >> >> > and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( >> >> >> >> Ahh, cell arrays, they bring back memories. Makes you pine for a >> >> dictionary, no? >> >> >> >> JDH >> > >> > Not to mention writeline, readline, string concatenation using +, >> English >> > wording of loops, list comprehension, etc., etc., etc. - if people only >> > knew... >> > >> > DG >> > >> >> >> >> -- >> > >> > Privacy is overrated; Identity isn't. >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Sun Aug 15 07:02:10 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 15 Aug 2010 13:02:10 +0200 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: <49251707-D525-460A-A995-F33E64C0367D@molden.no> Den 15. aug. 2010 kl. 04.34 skrev Gary Pajer : > Similar situation here. Been Matlab-free for about ten years, but > my new employer ... > Embed Python in a MEX an tell your employer you do MEX development. Works like a charm ;-) Or use Matlab's COM integration to run pywin32 as an ActiveX. Or use Matlab's JVM to run Jython. Just be creative, remember that employers' HR managers care a lot about buzzword compliance, as they don't understand the technology themselves. Sturla > I started looking at the code, and my stomach sank when I first read > > if (nargin = 1) > That's a bug by the way. > Oh, it hurts!! > > On Fri, Aug 13, 2010 at 4:41 PM, Benjamin Root > wrote: > @Josh: Awesome name. Very fitting... > > Another thing that I really love about matplotlib that drove me nuts > in Matlab was being unable to use multiple colormaps in the same > figure. > > And don't get me started with the map plotting tools and pcolor! > > Ben Root > > > On Fri, Aug 13, 2010 at 3:27 PM, Joshua Holbrook > wrote: > Ugh! I miss all that stuff anytime I have to use matlab. Working with > strings sucks especially hard. > > In fact, I've tried reimplementing half of the tools I wish I had. If > anyone cares: > > http://github.com/jesusabdullah/methlabs > > I'd love feedback. Perhaps ironically, I haven't used it much, due to > not wanting to give my matlabs dependencies. :/ > > --Josh > > > > On Fri, Aug 13, 2010 at 12:04 PM, David Goldsmith > wrote: > > > >> > After several years now of writing Python and now having > written my > >> > first > >> > on-the-job 15 operational MATLAB LOC, all of which are string, > cell > >> > array, > >> > and file processing, I'm ready to say: "MATLAB: what a PITA!" :-( > >> > >> Ahh, cell arrays, they bring back memories. Makes you pine for a > >> dictionary, no? > >> > >> JDH > > > > Not to mention writeline, readline, string concatenation using +, > English > > wording of loops, list comprehension, etc., etc., etc. - if people > only > > knew... > > > > DG > > > >> > >> -- > > > > Privacy is overrated; Identity isn't. > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jh at physics.ucf.edu Sun Aug 15 10:04:31 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Sun, 15 Aug 2010 10:04:31 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: While this thread is super off-topic (and long enough that I'm not going to quote it), I'm actually finding it very interesting, as a non-MATLAB person, to find out what I need to say to the MATLAB mafia here to demonstrate that MATLAB has sufficient flaws and non-followers "out in the real world" that perhaps certain classes could allow their students to use other languages, like Python. Could those contributing here put up a Cookbook page of "reasons why we've moved on from MATLAB", to be used as a resource by people trying to convince supervisors/professors/sponsors/clients that they should be allowed to use Python? It would be important for such a page to be polite and not offensive to those readers, who may like MATLAB. A similar page for IDL would be great....and did anyone notice that IDL 8.0 has a number of language enhancements, all designed to make it more like Python? Sadly, they fall well short. --jh-- From sturla at molden.no Sun Aug 15 13:43:46 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 15 Aug 2010 19:43:46 +0200 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> > Could those contributing here put up a Cookbook page of "reasons why > we've moved on from MATLAB", to be used as a resource by people trying > to convince supervisors/professors/sponsors/clients that they should > be allowed to use Python? There are many reasons, to name a few: 1. Ease of programming. Python is just a better language. Python is a general purpose language, MATLAB is restricted to simple numerical code. Python is easier to read (Matlab is more like Ruby or Lua). 2. Pass-by-value of arrays in Matlab. This basically means Matlab is unsuable for work with large data sets. Serious Matlab programming usually mean C programming with MEX wrappers. I find myself writing much more C or C++ when working with Matlab, which seriously impairs my productivity and creativity (it is easier to prototype and experiment in Python than C). 3. Better integration with C and Fortran in Python. Cython, ctypes and f2py is a lot easier than hand-coding CMEX and FMEX wrappers. C and Fortran integration is important for scientific computing. Also, Matlab does not play well with Fortran 95, just Fortran 77. For scientific computing, C, C++ and Fortran 77 are PITAs compared to Fortran 95. I even prefer Cython to the former three. 4. Matlab is single-treaded. Python allows threads, although there is a GIL (which can be freed when running C or Fortran library code). Matlab use MKL for multi-cores, NumPy/SciPy can be compiled against MKL or ACML. Threads are not just for parallel processing, but also for I/O and GUI interaction. 5. Heap-fragmentation makes long-running (32-bit) Matlab processes unstable. Instead of fixing the broken allocator, Matlab has a "pack" command that must be run e.g. once an hour. A long-running Python process can be trusted to complete if the code is stable, not so with Matlab in my experience. One has to save the state of the Matlab simulation/analysis regularly, and restart from where it stopped. 6. Matlab saves figures by taking a copy of the screen buffer. It is very annoying to discover that a long-running process has saved dozens of images of the screen saver, Windows' login screen, totally black images, or the mouse pointer. 7. Can we use OpenGL or VTK easily from Matlab? 8. GUI programming: Compare Matlab's rudimentary GUI support with wxPython or PyQt. 9. It is easy to use MPI from Python (e.g. mpi4py or just ctypes with the MPI library). The multiprocessing package also makes Python multiprocessing and IPC easy. Matlab has a huge fingerprint. Do you want multiple Matlab instances running in parallel? How much RAM do you have? 10. Memory-mapping is not usable in Matlab due to e.g. pass-by-value semantics, even though it is theoretically possible to memory map a file using a C MEX function. In Python we can memory map a file and alias the buffer with a NumPy array. 11. Database-support. Python has support for most major databases, and even has embedded dabases included in the standard library (e.g. bsddb and sqlite). Python supports HDF5 (PyTables and h5py). Efficient use of HDF5 is difficult from Matlab due to pass-by-value semantics. 12. Python costs less, even when subscribing to Enthough. 13. Matlab toolboxes cost extra. There is no extra fee to use scipy.signal and scipy.stats. 14. Google and NASA use Python. 15. It is easier to teach Python to students. My university (Oslo) use Python as teaching language for that reason. That is, with this textbook written for teaching Python to science students: http://books.google.no/books?id=cVof07z_rA4C&printsec=frontcover#v=onepage&q&f=false 16. Matplotlib makes nicer figures (e.g. antialiased rendering). 17. It is easy to spawn processes from Python and communicate with pipes. Thus we can use external programs easily from Python. 18. Python's standard library has support for almost any thinkable problem. The number of external libraries is immense. 19. Another memory issue: NumPy does not make a copy when creating a view. We don't need to copy data to reference strided array sections or subarrays. NumPy works like Fortran 95 pointers, which is very memory efficient and expressive for numerical work. 20. Etc. (I could go on and on, but have to stop somewhere.) Some strong points for Matlab: 1. The IDE, debugger, and interactive prompt. ipython cannot compare to Matlab. 2. FFTW. 3. Java VM. 4. More users. 5. Better documentation. > A similar page for IDL would be great....and did anyone notice that > IDL 8.0 has a number of language enhancements, all designed to make it > more like Python? Sadly, they fall well short. IDL feels too much like FORTRAN IV, whcih by the way is an F-word. Sturla From jsseabold at gmail.com Sun Aug 15 14:00:38 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Sun, 15 Aug 2010 14:00:38 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> Message-ID: On Sun, Aug 15, 2010 at 1:43 PM, Sturla Molden wrote: > >> Could those contributing here put up a Cookbook page of "reasons why >> we've moved on from MATLAB", to be used as a resource by people trying >> to convince supervisors/professors/sponsors/clients that they should >> be allowed to use Python? > > There are many reasons, to name a few: > > 1. Ease of programming. Python is just a better language. Python is a > general purpose language, MATLAB is restricted to simple numerical code. > Python is easier to read (Matlab is more like Ruby or Lua). > > 2. Pass-by-value of arrays in Matlab. This basically means Matlab is > unsuable for work with large data sets. Serious Matlab programming usually > mean C programming with MEX wrappers. I find myself writing much more C or > C++ when working with Matlab, which seriously impairs my productivity and > creativity (it is easier to prototype and experiment in Python than C). > > 3. Better integration with C and Fortran in Python. Cython, ctypes and > f2py is a lot easier than hand-coding CMEX and FMEX wrappers. C and > Fortran integration is important for scientific computing. Also, Matlab > does not play well with Fortran 95, just Fortran 77. For scientific > computing, C, C++ and Fortran 77 are PITAs compared to Fortran 95. I even > prefer Cython to the former three. > > 4. Matlab is single-treaded. Python allows threads, although there is a > GIL (which can be freed when running C or Fortran library code). Matlab > use MKL for multi-cores, NumPy/SciPy can be compiled against MKL or ACML. > Threads are not just for parallel processing, but also for I/O and GUI > interaction. > > 5. Heap-fragmentation makes long-running (32-bit) Matlab processes > unstable. Instead of fixing the broken allocator, Matlab has a "pack" > command that must be run e.g. once an hour. A long-running Python process > can be trusted to complete if the code is stable, not so with Matlab in my > experience. One has to save the state of the Matlab simulation/analysis > regularly, and restart from where it stopped. > > 6. Matlab saves figures by taking a copy of the screen buffer. It is very > annoying to discover that a long-running process has saved dozens of > images of the screen saver, Windows' login screen, totally black images, > or the mouse pointer. > > 7. Can we use OpenGL or VTK easily from Matlab? > > 8. GUI programming: Compare Matlab's rudimentary GUI support with wxPython > or PyQt. > > 9. It is easy to use MPI from Python (e.g. mpi4py or just ctypes with the > MPI library). The multiprocessing package also makes Python > multiprocessing and IPC easy. Matlab has a huge fingerprint. Do you want > multiple Matlab instances running in parallel? How much RAM do you have? > > 10. Memory-mapping is not usable in Matlab due to e.g. pass-by-value > semantics, even though it is theoretically possible to memory map a file > using a C MEX function. In Python we can memory map a file and alias the > buffer with a NumPy array. > > 11. Database-support. Python has support for most major databases, and > even has embedded dabases included in the standard library (e.g. bsddb and > sqlite). Python supports HDF5 (PyTables and h5py). Efficient use of HDF5 > is difficult from Matlab due to pass-by-value semantics. > > 12. Python costs less, even when subscribing to Enthough. > > 13. Matlab toolboxes cost extra. There is no extra fee to use scipy.signal > and scipy.stats. The cost and the licensing headaches I have had with MATLAB in our labs and with my student version (32-bit only though they don't tell you this until after purchase that I saw, one machine only, and one OS only even if it's on the same machine/same user) have really turned me off from using it even though it is more or less the standard for macroeconomics at the moment. I also found getting my 32-bit MATLAB to compile C extensions on a 64-bit system/software to be a PITA (somewhat related, I still haven't been able to install mlabwrap), but maybe that's just my lack of knowledge. > > 14. Google and NASA use Python. > > 15. It is easier to teach Python to students. My university (Oslo) use > Python as teaching language for that reason. That is, with this textbook > written for teaching Python to science students: > > http://books.google.no/books?id=cVof07z_rA4C&printsec=frontcover#v=onepage&q&f=false And it's a no-brainer to go to MATLAB after learning the ins and outs of Python. Python has a much better community for beginners (any level really) I think. Python gives you the skill set to work in other languages and environments while the converse is not necessarily true. > > 16. Matplotlib makes nicer figures (e.g. antialiased rendering). > > 17. It is easy to spawn processes from Python and communicate with pipes. > Thus we can use external programs easily from Python. > > 18. Python's standard library has support for almost any thinkable > problem. The number of external libraries is immense. > > 19. Another memory issue: NumPy does not make a copy when creating a view. > We don't need to copy data to reference strided array sections or > subarrays. NumPy works like Fortran 95 pointers, which is very memory > efficient and expressive for numerical work. > > 20. Etc. (I could go on and on, but have to stop somewhere.) > > > Some strong points for Matlab: > > 1. The IDE, debugger, and interactive prompt. ipython cannot compare to > Matlab. Though Spyder is coming along nicely. > > 2. FFTW. > > 3. Java VM. > > 4. More users. > > 5. Better documentation. > And as Josef mention, the file exchange is nice, though a lot of the code is BSD licensed. My $.02, Skipper > > >> A similar page for IDL would be great....and did anyone notice that >> IDL 8.0 has a number of language enhancements, all designed to make it >> more like Python? ?Sadly, they fall well short. > > IDL feels too much like FORTRAN IV, whcih by the way is an F-word. > > > > > > Sturla > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From waterbug at pangalactic.us Sun Aug 15 14:19:17 2010 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Sun, 15 Aug 2010 14:19:17 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> Message-ID: <4C682FA5.40606@pangalactic.us> [scipy/python vs. matlab discussion elided] To all potential future contributors to this thread: At the risk of adding noise, I'd like to suggest that if you're going to create an extended discussion thread (even if it is a worthwhile hijack, as this one is, I think), at least change its name to something that summarizes the discussion topic! Maybe something like "scipy/python vs. matlab" ... Steve From josef.pktd at gmail.com Sun Aug 15 15:40:17 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 15 Aug 2010 15:40:17 -0400 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> Message-ID: On Sun, Aug 15, 2010 at 1:43 PM, Sturla Molden wrote: > >> Could those contributing here put up a Cookbook page of "reasons why >> we've moved on from MATLAB", to be used as a resource by people trying >> to convince supervisors/professors/sponsors/clients that they should >> be allowed to use Python? > > There are many reasons, to name a few: > > 1. Ease of programming. Python is just a better language. Python is a > general purpose language, MATLAB is restricted to simple numerical code. > Python is easier to read (Matlab is more like Ruby or Lua). Organizing the code for a project in matlab is a lot more difficult than in python, modules, classes, datastructures besides structs are missing, weaker or messier. I find numerical code easy to read and write in matlab. > > 2. Pass-by-value of arrays in Matlab. This basically means Matlab is > unsuable for work with large data sets. Serious Matlab programming usually > mean C programming with MEX wrappers. I find myself writing much more C or > C++ when working with Matlab, which seriously impairs my productivity and > creativity (it is easier to prototype and experiment in Python than C). It's not fully pass-by-value, a copy is only made if the matrix/array is changed, but not if only accessed for getting the data. And from the comments I have seen they keep improving lazy copying. > > 3. Better integration with C and Fortran in Python. Cython, ctypes and > f2py is a lot easier than hand-coding CMEX and FMEX wrappers. C and > Fortran integration is important for scientific computing. Also, Matlab > does not play well with Fortran 95, just Fortran 77. For scientific > computing, C, C++ and Fortran 77 are PITAs compared to Fortran 95. I even > prefer Cython to the former three. > > 4. Matlab is single-treaded. Python allows threads, although there is a > GIL (which can be freed when running C or Fortran library code). Matlab > use MKL for multi-cores, NumPy/SciPy can be compiled against MKL or ACML. > Threads are not just for parallel processing, but also for I/O and GUI > interaction. since 2007a: "MATLAB has multithreaded computation support for many linear algebra and element-wise numeric operations, allowing performance improvement on multicore and multiprocessor systems." > > 5. Heap-fragmentation makes long-running (32-bit) Matlab processes > unstable. Instead of fixing the broken allocator, Matlab has a "pack" > command that must be run e.g. once an hour. A long-running Python process > can be trusted to complete if the code is stable, not so with Matlab in my > experience. One has to save the state of the Matlab simulation/analysis > regularly, and restart from where it stopped. > > 6. Matlab saves figures by taking a copy of the screen buffer. It is very > annoying to discover that a long-running process has saved dozens of > images of the screen saver, Windows' login screen, totally black images, > or the mouse pointer. I never heard of or seen this > > 7. Can we use OpenGL or VTK easily from Matlab? > > 8. GUI programming: Compare Matlab's rudimentary GUI support with wxPython > or PyQt. > > 9. It is easy to use MPI from Python (e.g. mpi4py or just ctypes with the > MPI library). The multiprocessing package also makes Python > multiprocessing and IPC easy. Matlab has a huge fingerprint. Do you want > multiple Matlab instances running in parallel? How much RAM do you have? I found the parallel processing toolbox (mainly using parfor) relatively easy to use to keep four cores busy with some lengthy nested optimization loops. > > 10. Memory-mapping is not usable in Matlab due to e.g. pass-by-value > semantics, even though it is theoretically possible to memory map a file > using a C MEX function. In Python we can memory map a file and alias the > buffer with a NumPy array. > > 11. Database-support. Python has support for most major databases, and > even has embedded dabases included in the standard library (e.g. bsddb and > sqlite). Python supports HDF5 (PyTables and h5py). Efficient use of HDF5 > is difficult from Matlab due to pass-by-value semantics. I never needed it, but with odbc and jdbc support and and http://developer.berlios.de/projects/mksqlite/ database support shouldn't be so bad. http://www.mathworks.com/access/helpdesk/help/toolbox/database/ > > 12. Python costs less, even when subscribing to Enthough. > > 13. Matlab toolboxes cost extra. There is no extra fee to use scipy.signal > and scipy.stats. > > 14. Google and NASA use Python. > > 15. It is easier to teach Python to students. My university (Oslo) use > Python as teaching language for that reason. That is, with this textbook > written for teaching Python to science students: > > http://books.google.no/books?id=cVof07z_rA4C&printsec=frontcover#v=onepage&q&f=false > > 16. Matplotlib makes nicer figures (e.g. antialiased rendering). > > 17. It is easy to spawn processes from Python and communicate with pipes. > Thus we can use external programs easily from Python. > > 18. Python's standard library has support for almost any thinkable > problem. The number of external libraries is immense. > > 19. Another memory issue: NumPy does not make a copy when creating a view. > We don't need to copy data to reference strided array sections or > subarrays. NumPy works like Fortran 95 pointers, which is very memory > efficient and expressive for numerical work. > > 20. Etc. (I could go on and on, but have to stop somewhere.) > > > Some strong points for Matlab: > > 1. The IDE, debugger, and interactive prompt. ipython cannot compare to > Matlab. > > 2. FFTW. > > 3. Java VM. > > 4. More users. More freely available packages in some (scientific) areas/fields. (commercial) toolboxes are available for many areas. > > 5. Better documentation. My impression is that matlab is improving pretty fast, I have seen big improvements between 2006a and 2009b or 2010a stats, econometrics toolboxes, parallel support, new classes. Some of the criticism of an old matlab might not be accurate anymore. and matlab is doing pretty well http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html I feel I have to defend MATLAB, because besides GAUSS (and of course numpy), it's the best matrix/array language, which is very nice for writing econometrics/statistics code (and I find it much more readable than R, Stata or SAS code). Josef > > >> A similar page for IDL would be great....and did anyone notice that >> IDL 8.0 has a number of language enhancements, all designed to make it >> more like Python? ?Sadly, they fall well short. > > IDL feels too much like FORTRAN IV, whcih by the way is an F-word. > > > > > > Sturla > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sturla at molden.no Sun Aug 15 16:10:13 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 15 Aug 2010 22:10:13 +0200 Subject: [Numpy-discussion] Matlab vs. Python (Was: Re: [SciPy-Dev] Good-bye, sort of (John Hunter)) In-Reply-To: References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> Message-ID: <2f4c5f09d10a7a78684f2f15461fcaa6.squirrel@webmail.uio.no> > It's not fully pass-by-value, a copy is only made if the matrix/array > is changed, but not if only accessed for getting the data. And from > the comments I have seen they keep improving lazy copying. Yes it is copy-on-write. But that means we must restrict ourselves to read-only access. It also means that we cannot memory map a file. If there is a write operation, we get a copy of the whole file into RAM. Matlab also takes a copy if we create an array slice. It means for example that a wavelet transform written in Python will be O(n) with respect to memory, whereas it will be O(n log n) in Matlab. I am working on data that are recorded at 48 kHz, 16-32 channels, for several minutes. Memory issues like these do matter. >> 4. Matlab is single-treaded. Python allows threads, although there is a >> GIL (which can be freed when running C or Fortran library code). Matlab >> use MKL for multi-cores, NumPy/SciPy can be compiled against MKL or >> ACML. >> Threads are not just for parallel processing, but also for I/O and GUI >> interaction. > > since 2007a: > "MATLAB has multithreaded computation support for many linear algebra > and element-wise numeric operations, allowing performance improvement > on multicore and multiprocessor systems." That is due to MKL and FFTW. I have NumPy and SciPy linked with MKL. It uses all four cores on my laptop for linear algebra as well. Matlab does not have a multi-threaded interpreter. Please don't confuse this. Sturla From sturla at molden.no Sun Aug 15 16:27:34 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 15 Aug 2010 22:27:34 +0200 Subject: [Numpy-discussion] Matlab vs. Python (Was: Re: [SciPy-Dev] Good-bye, sort of (John Hunter)) In-Reply-To: <2f4c5f09d10a7a78684f2f15461fcaa6.squirrel@webmail.uio.no> References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> <2f4c5f09d10a7a78684f2f15461fcaa6.squirrel@webmail.uio.no> Message-ID: > Matlab also takes a copy if we create an array slice. It means for example > that a wavelet transform written in Python will be O(n) with respect to > memory, whereas it will be O(n log n) in Matlab. Other examples: A reshape will create a new array in Matlab. It will create a view in NumPy. A transpose will create a new array in Matlab and be O(m*n). A transpose will just create a view in NumPy (i.e. reverse stride and shape attributes) and be O(1). Matlab does not broadcast. We must explicitely match array shapes using repmat, which by the way creates a new array. NumPy allows arrays to be updated inplace. Matlab always creates a new copy (Matlab arrays are immutable). From alexlz at lmn.pub.ro Sun Aug 15 16:51:00 2010 From: alexlz at lmn.pub.ro (Ioan-Alexandru Lazar) Date: Sun, 15 Aug 2010 23:51:00 +0300 Subject: [Numpy-discussion] Matlab vs. Python (Was: Re: [SciPy-Dev] Good-bye, sort of (John Hunter)) (Sturla Molden) In-Reply-To: References: Message-ID: <6fd8f6781fa6cb1cb2d049c83e2054a6.squirrel@wm.lmn.pub.ro> Hi everyone, I've been pretty happy with how Spider went along when I tried it. I use Emacs but I think Spyder is perfectly usable for someone used to Matlab. A few HPC-centric of my reasons (I've shamelessly copy-pasted this because I'm lazy right now): 1. Python is an expressive, full-fledged, general-purpose application language. There is slightly more boilerplate for doing math-related operations (i.e. creating a matrix is not as simple as A=[[1, 2, 3]; [4, 5, 6]] but rather A=numpy.array([[1,2,3],[4,5,6]])). On the other hand, everything other than math can be expressed without causing severe nausea and vomiting to the user. 2. The package-module structure of Python allows me to obtain a significantly less cluttered workspace. 3. There?s built-in support for automatically generating meaningful and useful documentation 4. The wonderful FFI support allows me to easily work with external C code. Writing MEX functions is a total mess, I hate that. On the other hand, it?s easy for me to integrate Python with a custom-compiled version of UMFPACK or any other solver, and wrappers can be automatically generated with SWIG for a minimal amount of effort. 5. There is a Matlab wrapper called mlabwrap, so legacy code written in Matlab is not lost effort. 6. I can use Emacs for my development rather than choosing between a) working in half-assed environment without code completion or b) working with Matlab?s incredibly slow and sloppy user interface on Unix systems. 7. I have built-in support for primitives like linked lists, queues, stacks and tuples 8. I have standards-compliant support for MPI that does not look alien in Python (there?s support for that in Matlab too, but it feels like you?re coding on Mars in a library written somewhere in the Andromeda galaxy). This is extremely important to me right now because the algorithm we?re implementing will be presented in a conference in September, and I want to have an implementation that uses standard tools, available to every HPC developer. On the other hand, I will be able to further optimize communication (if this will seem to be required) by using a good distributed-object library like PyRO. Some details about what exactly it is that I am using it for are on my blog at http://zencoding.org/archives/137#more-137, although it's rather sketchy -- I plan to write a slightly more detailed document about my experience with Python for HPC and how it compares against Matlab's PCS and DCS. I've been banging my head against Matlab for a while so I'll gladly write a thing or two on the wiki if you think this sort of use case is relevant. One other observation -- Matlab does have some support for parallel processing via PCS, but you have to pay for that, and it's not too flexible. There's also DCS for distributed computing. Some of my colleagues have been using those and aren't too happy about their flexibility just yet. MathWorks is also not too verbose about what particular parts of Matlab are parallelized and not, so we've been randomly stumbling upon parts that were actually parallelized (e.g. UMFPACK is linked against a multithreaded BLAS) even though we thought they weren't. Their documentation is detailed, but as far as careful details about under-the-hood issues and optimization methods, it's a bad joke. If you need it as a glorified handheld calculator or as a prototyping tool, it's great, but writing full-fledged apps in it is painful. Best regards, Alexandru Lazar, Numerical Modeling Laboratory, Politehnica University of Bucharest From josef.pktd at gmail.com Sun Aug 15 21:53:47 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 15 Aug 2010 21:53:47 -0400 Subject: [Numpy-discussion] Matlab vs. Python (Was: Re: [SciPy-Dev] Good-bye, sort of (John Hunter)) In-Reply-To: References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> <2f4c5f09d10a7a78684f2f15461fcaa6.squirrel@webmail.uio.no> Message-ID: On Sun, Aug 15, 2010 at 4:27 PM, Sturla Molden wrote: > >> Matlab also takes a copy if we create an array slice. It means for example >> that a wavelet transform written in Python will be O(n) with respect to >> memory, whereas it will be O(n log n) in Matlab. > > Other examples: > > A reshape will create a new array in Matlab. It will create a view in NumPy. > > A transpose will create a new array in Matlab and be O(m*n). A transpose > will just create a view in NumPy (i.e. reverse stride and shape > attributes) and be O(1). > > Matlab does not broadcast. We must explicitely match array shapes using > repmat, which by the way creates a new array. I agree broadcasting is one of my favorite features of numpy compared to any other array language I know > > NumPy allows arrays to be updated inplace. Matlab always creates a new > copy (Matlab arrays are immutable). inplace operation in matlab: http://blogs.mathworks.com/loren/2007/03/22/in-place-operations-on-data/ not as fancy but improving matlab has JIT compiler also this for some mostly balanced arguments, I think, http://stackoverflow.com/questions/179904/what-is-matlab-good-for-why-is-it-so-used-by-universities-when-is-it-better-tha I don't have very large memory requirements, and I didn't do enough similar things in matlab and python to have experience with these differences. Josef > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josh.holbrook at gmail.com Sun Aug 15 23:26:35 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Sun, 15 Aug 2010 19:26:35 -0800 Subject: [Numpy-discussion] Matlab vs. Python (Was: Re: [SciPy-Dev] Good-bye, sort of (John Hunter)) In-Reply-To: References: <0605a583477f0ae77e9744a7d10ef838.squirrel@webmail.uio.no> <2f4c5f09d10a7a78684f2f15461fcaa6.squirrel@webmail.uio.no> Message-ID: On Sun, Aug 15, 2010 at 5:53 PM, wrote: > On Sun, Aug 15, 2010 at 4:27 PM, Sturla Molden wrote: >> >>> Matlab also takes a copy if we create an array slice. It means for example >>> that a wavelet transform written in Python will be O(n) with respect to >>> memory, whereas it will be O(n log n) in Matlab. >> >> Other examples: >> >> A reshape will create a new array in Matlab. It will create a view in NumPy. >> >> A transpose will create a new array in Matlab and be O(m*n). A transpose >> will just create a view in NumPy (i.e. reverse stride and shape >> attributes) and be O(1). >> >> Matlab does not broadcast. We must explicitely match array shapes using >> repmat, which by the way creates a new array. > > I agree broadcasting is one of my favorite features of numpy compared > to any other array language I know > >> >> NumPy allows arrays to be updated inplace. Matlab always creates a new >> copy (Matlab arrays are immutable). > > inplace operation in matlab: > http://blogs.mathworks.com/loren/2007/03/22/in-place-operations-on-data/ > not as fancy but improving > > matlab has JIT compiler > > also this for some mostly balanced arguments, I think, > http://stackoverflow.com/questions/179904/what-is-matlab-good-for-why-is-it-so-used-by-universities-when-is-it-better-tha > > I don't have very large memory requirements, and I didn't do enough > similar things in matlab and python to have experience with these > differences. > > Josef > >> >> >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > This bit from the SO answers: > Something you have to realize about MATLAB is that it started off as a wrapper on Fortran > libraries for linear algebra. For a long time, it had an attitude that "all the world is an array of > doubles (floats)". As a language, it has grown very organically, an there are some flaws that > are very much baked in, if you look at it just as a programming language. rings very true to me. On the other hand: > There is a large community of users that share numerical code I have never really been satisfied with Matlab Central, vs. Pypi, Github, Sourceforge, and all that. Then again, I've been spoiled by the OSS community and have some pretty specific needs for MATLAB, so maybe I'm missing something. --Josh From asmund.hjulstad at gmail.com Mon Aug 16 03:51:39 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Mon, 16 Aug 2010 10:51:39 +0300 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran Message-ID: Hi all, Are there any success stories in building f2py extensions on AMD64 with latest versions? Building the same extension on 32 bit works like a charm. I am having trouble finding documentation or examples, is it supposed to be working? Compiling (with distutils) works like a charm, but that does not help when I'm stuck figuring out link dependencies. It seems to me that the gfortran library depends on a mingw that is in conflict with some CRT library. I have to admit, I'm probably in way too deep waters, and should really be satisfied with 32 bit, but still, it would be fun to get it working. GFortran is from MinGW-w64 project on sourceforge, version 4.5.1 prerelease. Any pointers or other experiences? Regards, ?smund Hjulstad -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph at debian.org Mon Aug 16 04:50:48 2010 From: morph at debian.org (Sandro Tosi) Date: Mon, 16 Aug 2010 10:50:48 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: Hi all, sorry for the delay On Sun, Aug 1, 2010 at 18:38, Ralf Gommers wrote: > I am pleased to announce the availability of the first beta of NumPy 1.5.0. > This will be the first NumPy release to include support for Python 3, as > well as for Python 2.7. Please try this beta and report any problems on the > NumPy mailing list. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > Please note that binaries for Python 3.1 are not yet up, they will follow as > soon as a minor issue with building them is resolved. Building from source > with Python 3.1 should work without problems. I gave it a run on the Debian packaging system and these are the results: - python 3.1 can't compile it: $ python3 setup.py build Traceback (most recent call last): File "setup.py", line 210, in setup_package() File "setup.py", line 174, in setup_package import py3tool ImportError: No module named py3tool but this is already known, py3tool is missing from the tarball. - python2.6 build, installation and numpy.test() works fine - I have a problem building documentation: # build doc only for default python version (export MPLCONFIGDIR=. ; make -C doc html PYTHONPATH=../build/lib.linux-x86_64-2.6) make[1]: Entering directory `/tmp/buildd/python-numpy-1.5.0~b1/doc' mkdir -p build python \ ./sphinxext/autosummary_generate.py source/reference/*.rst \ -p dump.xml -o source/reference/generated ./sphinxext/autosummary_generate.py:18: DeprecationWarning: The numpydoc.autosummary extension can also be found as sphinx.ext.autosummary in Sphinx >= 0.6, and the version in Sphinx >= 0.7 is superior to the one in numpydoc. This numpydoc version of autosummary is no longer maintained. from autosummary import import_by_name Failed to import 'numpy.__array_priority__': Failed to import 'numpy.core.defchararray.len': Failed to import 'numpy.generic.__squeeze__': touch build/generate-stamp mkdir -p build/html build/doctrees LANG=C sphinx-build -b html -d build/doctrees source build/html Running Sphinx v0.6.6 Extension error: Could not import extension numpydoc (exception: No module named domains.c) 1.5b1 1.5.0b1 I don't know exactly the reason for the "Failed to import X" but they are there also for 1.4.1 so they should not be a problem. Anyhow, there is no file named 'domains.c' in the tarball. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From v.gkinis at nbi.ku.dk Mon Aug 16 05:12:38 2010 From: v.gkinis at nbi.ku.dk (Vasileios Gkinis) Date: Mon, 16 Aug 2010 11:12:38 +0200 Subject: [Numpy-discussion] f2py performance question from a rookie Message-ID: <4C690106.80407@nbi.ku.dk> An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Aug 16 05:14:43 2010 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Aug 2010 11:14:43 +0200 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: References: Message-ID: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> Two criticial import libraries for mingw-w64 are missing (libpython26.a and msvcr90.a). We cannot build C extensions for Python with mingw-w64. If you can build with disutils, your are not using mingw-w64 but another C compiler. There are rumors of incompatibility issues between libgfortran and the CRT or SciPy on Windows 64. David Cournapeau might know the details. It seems f2py on Windows 64 requires a commercial Fortran compiler (the ones worth looking at are Absoft, Intel and PGI) and e.g. Microsoft's free C/C++ compiler (download Windows 7 SDK for .NET 3.5 -- not any later or earlier version). You need to have the environment variables DISTUTILS_USE_SDK and MSSdk set to build with the free SDK compiler. f2py will enforce the former, the latter comes from running setenv from the SDK. The latter is important as you will need to set up the environments for the Fortran and the C compiler in the same command window before running f2py. I also had to remove support for Compaq Visual Fortran from the f2py source to make anything build, as it crashes f2py on Windows 64 (this does not happen on 32-bit Windows). Building f2py extensions on Windows 64 is a bit more tricky. And I've had no luck with gfortran so far. Sturla > Hi all, > > Are there any success stories in building f2py extensions on AMD64 with > latest versions? Building the same extension on 32 bit works like a charm. > > I am having trouble finding documentation or examples, is it supposed to > be > working? > > Compiling (with distutils) works like a charm, but that does not help when > I'm stuck figuring out link dependencies. It seems to me that the gfortran > library depends on a mingw that is in conflict with some CRT library. > > I have to admit, I'm probably in way too deep waters, and should really be > satisfied with 32 bit, but still, it would be fun to get it working. > > GFortran is from MinGW-w64 project on sourceforge, version 4.5.1 > prerelease. > > Any pointers or other experiences? > > Regards, > ?smund Hjulstad > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robince at gmail.com Mon Aug 16 06:05:37 2010 From: robince at gmail.com (Robin) Date: Mon, 16 Aug 2010 11:05:37 +0100 Subject: [Numpy-discussion] f2py performance question from a rookie In-Reply-To: <4C690106.80407@nbi.ku.dk> References: <4C690106.80407@nbi.ku.dk> Message-ID: On Mon, Aug 16, 2010 at 10:12 AM, Vasileios Gkinis wrote: > Hi all, > > This is a question on f2py. > I am using a Crank Nicholson scheme to model a diffusion process and in the > quest of some extra speed I started playing with f2py. > However I do not seem to be able to get any significant boost in the > performance of the code. > Try adding -DF2PY_REPORT_ON_ARRAY_COPY to the f2py command line. This will cause f2py to report any array copies. If any of the types/ordering of the arrays don't match f2py will silently make a copy - this can really affect performance. Cheers Robin > > In the following simple example I am building a tridiagonal array with > plain python for loops and by calling a simple fortran subroutine that does > the same thing. > > Here goes the python script: > > import numpy as np > import time > import learnf90_05 > import sys > > run = sys.argv[1] > > try: > dim = np.float(sys.argv[2]) > except: > dim = 500 > elem = np.ones(dim) > > ### Python routine > > if run == "Python": > t_python_i = time.time() > array_python = np.zeros((dim, dim)) > for row in np.arange(1,dim-1): > array_python[row, row-1] = elem[row] > array_python[row, row] = elem[row] > array_python[row,row+1] = elem[row] > > t_python_f = time.time() > python_time = t_python_f - t_python_i > print("Python time: %0.5e" %(python_time)) > > ###fortran routine > > elif run == "Fortran": > t_fortran_i = time.time() > fortran_array = learnf90_05.test(j = dim, element = elem) > t_fortran_f = time.time() > fortran_time = t_fortran_f - t_fortran_i > print("Fortran time: %0.5e" %(fortran_time)) > > And the fortran subroutine called test is here: > > subroutine test(j, element, a) > integer, intent(in) :: j > integer :: row, col > real(kind = 8), intent(in) :: element(j) > real(kind = 8), intent(out) :: a(j,j) > > do row=2,j-1 > a(row,row-1) = element(row) > a(row, row) = element(row) > a(row, row+1) = element(row) > enddo > > end > > The subroutine is compiled with Gfortran 4.2 on Osx 10.5.8. > > Running the python script with array sizes 300x300 and 3000x3000 gives for > the python plain for loops: > > dhcp103:learnf vasilis$ python fill_array.py Python 3000 > Python time: 1.56063e-01 > dhcp103:learnf vasilis$ python fill_array.py Python 300 > Python time: 4.82297e-03 > > and for the fortran subroutine: > > dhcp103:learnf vasilis$ python fill_array.py Fortran 3000 > Fortran time: 1.16298e-01 > dhcp103:learnf vasilis$ python fill_array.py Fortran 300 > Fortran time: 8.83102e-04 > > It looks like the gain in performance is rather low compared to tests i > have seen elsewhere. > > Am I missing something here..? > > Cheers...Vasilis > > > -- > ------------------------------------------------ > Vasileios Gkinis PhD student > Center for Ice and Climate > Niels Bohr Institute > Juliane Maries Vej 30 > 2100 Copenhagen > Denmark > email: v.gkinis at nbi.ku.dk skype: vasgkin Tel: +45 > 353 25913 ------------------------------------------------ > -- > ------------------------------------------------ > Vasileios Gkinis PhD student > Center for Ice and Climate > Niels Bohr Institute > Juliane Maries Vej 30 > 2100 Copenhagen > Denmark > email: v.gkinis at nbi.ku.dk skype: vasgkin Tel: +45 > 353 25913 ------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scipy at samueljohn.de Mon Aug 16 06:17:13 2010 From: scipy at samueljohn.de (Samuel John) Date: Mon, 16 Aug 2010 12:17:13 +0200 Subject: [Numpy-discussion] Building numpy on Mac OS X 10.6, i386 (no ppc) 32/64bit: Error in Fortran tests due to ppc64 Message-ID: Hello! At first, I'd like to say thanks to the numpy/scipy team and all contributors. Great software! On Snow Leopard, aka Mac OS X 10.6.4 (server) I managed to build numpy 2.0.0.dev8636 (and scipy 0.9.0.dev6646) for arch i386 in combined 32/64bit against MacPorts python27 (No ppc here!). All tests pass (yeha!), except for the fortran related ones. I think there is an issue with detecting the right arch. My numpy and python are both i386 32/64 bit but now ppc. Only these tests fail, all others pass: test_callback.TestF77Callback.test_all ... ERROR test_mixed.TestMixed.test_all ... ERROR test_return_character.TestF77ReturnCharacter.test_all ... ERROR test_return_character.TestF90ReturnCharacter.test_all ... ERROR test_return_complex.TestF77ReturnComplex.test_all ... ERROR test_return_complex.TestF90ReturnComplex.test_all ... ERROR test_return_integer.TestF77ReturnInteger.test_all ... ERROR test_return_integer.TestF90ReturnInteger.test_all ... ERROR test_return_logical.TestF77ReturnLogical.test_all ... ERROR test_return_logical.TestF90ReturnLogical.test_all ... ERROR test_return_real.TestCReturnReal.test_all ... ok test_return_real.TestF77ReturnReal.test_all ... ERROR test_return_real.TestF90ReturnReal.test_all ... ERROR [...] ---------------------------------------------------------------------- Ran 2989 tests in 47.008s FAILED (KNOWNFAIL=4, SKIP=1, errors=12) Some more information (Perhaps I did some known mistake in those steps? Details at the end of this mail): o Mac OS X 10.6.4 (intel Core 2 duo) o Python 2.7 (r27:82500, Aug 15 2010, 12:19:40) [GCC 4.2.1 (Apple Inc. build 5659) + GF 4.2.4] on darwin o gcc --version i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664) o gfortran --version GNU Fortran (GCC) 4.2.1 (Apple Inc. build 5659) + GF 4.2.4 from gfortran from http://r.research.att.com/tools/ o I used the BLAS/LAPACK that is provided by Apple's Accelerate framework. o environment: export CFLAGS="-arch i386 -arch x86_64" export FFLAGS="-m32 -m64" export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -framework Accelerate" o bulid: python setup.py build --fcompiler=gnu95 I have not found a matching ticket in trac. Should I open one or did I something very stupid during the build process? Thanks! Samuel PS: I failed to succeed in the first shot with python.org's official fat precompiled .dmg-file release (ppc/i386 32/64 bit), so I used MacPorts. Later today, I'll try again to compile against python.org because I think numpy/scipy recommends that version. For completeness, here are my build steps: o Building numpy/scipy from source: http://scipy.org/Installing_SciPy/Mac_OS_X: - Make sure XCode is installed with the Development target 10.4 SDK - Download and install gfortran from http://r.research.att.com/tools/ - svn co http://svn.scipy.org/svn/numpy/trunk numpy - svn co http://svn.scipy.org/svn/scipy/trunk scipy - sudo port install fftw-3 - sudo port install suitesparse - sudo port install swig-python - mkdir scipy_numpy; cd scipy_numpy - cd numpy - cp site.cfg.example site.cfg - You may want to copy the site.cfg to ~/.numpy-site.cfg - Edit site.cfg to contain only the following: [DEFAULT] library_dirs = /opt/local/lib include_dirs = /opt/local/include [amd] amd_libs = amd [umfpack] umfpack_libs = umfpack [fftw] libraries = fftw3 - export MACOSX_DEPLOYMENT_TARGET=10.6 - export CFLAGS="-arch i386 -arch x86_64" - export FFLAGS="-m32 -m64" - export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -framework Accelerate" - export PYTHONPATH="/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/" - python setup.py build --fcompiler=gnu95 - sudo python setup.py install - cd .. - cd scipy - sed 's|include <\(umfpack[^\.]*\.h\)>|include |g' scipy/sparse/linalg/dsolve/umfpack/___tmp.i - mv scipy/sparse/linalg/dsolve/umfpack/umfpack.i scipy/sparse/linalg/dsolve/umfpack/umfpack.old - mv scipy/sparse/linalg/dsolve/umfpack/___tmp.i scipy/sparse/linalg/dsolve/umfpack/umfpack.i - python setup.py build --fcompiler=gnu95 - cd - python import numpy; numpy.test() import scipy; scipy.test() A short excerpt of numpy.test()'s output: ====================================================================== ERROR: test_return_real.TestF90ReturnReal.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/case.py", line 367, in setUp try_run(self.inst, ('setup', 'setUp')) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/util.py", line 491, in try_run return func() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 341, in setUp module_name=self.module_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 73, in wrapper memo[key] = func(*a, **kw) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 156, in build_code module_name=module_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 73, in wrapper memo[key] = func(*a, **kw) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 128, in build_module % (cmd[4:], asstr(out))) RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', '/var/folders/sy/syxH3HBVEUyeFZOejvLLp+++-+2/-Tmp-/tmpJVSKpT/tmpHRbp1O.f90'] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building extension "_test_ext_module_5403" sources f2py options: [] f2py:> /var/folders/sy/syxH3HBVEUyeFZOejvLLp+++-+2/-Tmp-/tmpOAl6fb/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403module.c [?snip?] build_src: building npy-pkg config files running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize NAGFCompiler Could not locate executable f95 customize AbsoftFCompiler Could not locate executable f90 Could not locate executable f77 customize IBMFCompiler Could not locate executable xlf90 Could not locate executable xlf customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize GnuFCompiler Could not locate executable g77 customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using build_ext building '_test_ext_module_5403' extension compiling C sources C compiler: /usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes ortran f77 compiler: /usr/bin/gfortran [?snip?] Fortran f90 compiler: /usr/bin/gfortran [?snip?] Fortran fix compiler: /usr/bin/gfortran [?snip?] [?snip?] ld: warning: in /var/folders/xxx/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403module.o, file was built for unsupported file format which is not the architecture being linked (i386) [?snip?] ld: warning: in /usr/lib/libSystem.dylib, missing required architecture ppc64 in file ld: in /usr/lib/libSystem.B.dylib, missing required architecture ppc64 in file for architecture ppc64 collect2: ld returned 1 exit status lipo: can't open input file: /var/folders/sy/syxH3HBVEUyeFZOejvLLp+++-+2/-Tmp-//ccXwmeM6.out (No such file or directory) error: Command "/usr/bin/gfortran -Wall -arch ppc -arch i686 -arch x86_64 -arch ppc64 -Wall -undefined dynamic_lookup -bundle /var/folders/xxx/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403module.o /var/folders/xxx/src.macosx-10.6-x86_64-2.7/fortranobject.o /var/folders/sy/syxH3HBVEUyeFZOejvLLp+++-+2/-Tmp-/tmpOAl6fb/var/folders/sy/syxH3HBVEUyeFZOejvLLp+++-+2/-Tmp-/tmpJVSKpT/tmpHRbp1O.o /var/folders/xxx/src.macosx-10.6-x86_64-2.7/_test_ext_module_5403-f2pywrappers2.o -lgfortran -o ./_test_ext_module_5403.so" failed with exit status 1 [Note, i replace a long temp path by xxx just in this mail to make it more readable.] ---------------------------------------------------------------------- Ran 2989 tests in 47.008s FAILED (KNOWNFAIL=4, SKIP=1, errors=12) Btw: scipy.test(): Ran 5340 tests in 711.401s FAILED (KNOWNFAIL=14, SKIP=31, errors=3, failures=3) From scipy at samueljohn.de Mon Aug 16 06:44:29 2010 From: scipy at samueljohn.de (Samuel John) Date: Mon, 16 Aug 2010 12:44:29 +0200 Subject: [Numpy-discussion] Building numpy on Mac OS X 10.6, i386 (no ppc) 32/64bit: Error in Fortran tests due to ppc64 In-Reply-To: References: Message-ID: Perhaps related tickets, but no perfect match (as far as I can judge): - http://projects.scipy.org/numpy/ticket/1399 "distutils fails to build ppc64 support on Mac OS X when requested" This revision is older than the one I used, ergo should already be applied. - http://projects.scipy.org/numpy/ticket/1111 "Fix endianness-detection on ppc64 builds" closed. Already applied. - http://projects.scipy.org/numpy/ticket/527 "fortran linking flag option..." Perhaps that linking flag could help to tell numpy (distutils) the right arch? - http://projects.scipy.org/numpy/ticket/1170 "Possible Bug in F2PY Fortran Compiler Detection" Hmm, I don't know... Samuel From asmund.hjulstad at gmail.com Mon Aug 16 07:07:55 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Mon, 16 Aug 2010 14:07:55 +0300 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> Message-ID: 2010/8/16 Sturla Molden > Two criticial import libraries for mingw-w64 are missing (libpython26.a > and msvcr90.a). We cannot build C extensions for Python with mingw-w64. If > you can build with disutils, your are not using mingw-w64 but another C > compiler. > You are correct. The 32 bit setup is on a different computer, and there I'm gcc and gfortran built by equation solution (v4.5.0). (No MSVC, and it works well.) I was hoping to continue with gcc for 64bit, but understood that msvc was required. My attempted 64 bit setup is MSVC 2008 Pro for c, and mingw-w64 for the fortran part. Thank you for the information. I will have to look into options for a different fortran compiler, for now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.c.cherel at gmail.com Mon Aug 16 09:46:28 2010 From: guillaume.c.cherel at gmail.com (=?UTF-8?Q?Guillaume_Ch=C3=A9rel?=) Date: Mon, 16 Aug 2010 15:46:28 +0200 Subject: [Numpy-discussion] save data to csv with column names Message-ID: Hello, I'd like to know if there is an easy way to save a list of 1D arrays to a csv file, with the first line of the file being the column names. I found the following, but I can't get to save the column names: data = rec.array([X1,X2,X3,X4], names=[n1,n2,n3,n4]) savetxt("filename", data, delimiter=",", fmt=["%i","%d", "%f","%f"]) Thank you, Guillaume -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Aug 16 09:53:05 2010 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Aug 2010 15:53:05 +0200 Subject: [Numpy-discussion] f2py performance question from a rookie In-Reply-To: <4C690106.80407@nbi.ku.dk> References: <4C690106.80407@nbi.ku.dk> Message-ID: > It looks like the gain in performance is > rather low compared to tests i have seen elsewhere.
>
> Am I missing something here..?
>
> Cheers...Vasilis
Turn HTML off please. Use time.clock(), not time.time(). Try some tasks that actually takes a while. Tasks that take 10**-4 or 10**-3 seconds cannot be reliably timed on Windows or Linux. From jdh2358 at gmail.com Mon Aug 16 09:54:24 2010 From: jdh2358 at gmail.com (John Hunter) Date: Mon, 16 Aug 2010 08:54:24 -0500 Subject: [Numpy-discussion] save data to csv with column names In-Reply-To: References: Message-ID: 2010/8/16 Guillaume Ch?rel : > Hello, > I'd like to know if there is an easy way to save a list of 1D arrays to a > csv file, with the first line of the file being the column names. > > I found the following, but I can't get to save the column names: > > data = rec.array([X1,X2,X3,X4], names=[n1,n2,n3,n4]) > savetxt("filename", data, delimiter=",", fmt=["%i","%d", "%f","%f"]) import matplotlib.mlab as mlab mlab.rec2csv(data, 'myfile.csv') From jsseabold at gmail.com Mon Aug 16 09:54:37 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 16 Aug 2010 09:54:37 -0400 Subject: [Numpy-discussion] save data to csv with column names In-Reply-To: References: Message-ID: 2010/8/16 Guillaume Ch?rel : > Hello, > > I'd like to know if there is an easy way to save a list of 1D arrays to a > csv file, with the first line of the file being the column names. > > I found the following, but I can't get to save the column names: > > data = rec.array([X1,X2,X3,X4], names=[n1,n2,n3,n4]) > savetxt("filename", data, delimiter=",", fmt=["%i","%d", "%f","%f"]) > There is a patch to add this capability to savetxt. It was proposed to be included in Numpy 1.5, but I don't see that the patch was applied yet. You can use it for your stuff in the meantime. I've been using a version that I wrote since I need to do this quite often. http://projects.scipy.org/numpy/ticket/1079 I don't remember without looking if this patch also accepts recarrays and scrapes the names automatically. It should. Skipper From ralf.gommers at googlemail.com Mon Aug 16 11:36:14 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 16 Aug 2010 23:36:14 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: On Mon, Aug 16, 2010 at 4:50 PM, Sandro Tosi wrote: > Hi all, > sorry for the delay > > On Sun, Aug 1, 2010 at 18:38, Ralf Gommers > wrote: > > I am pleased to announce the availability of the first beta of NumPy > 1.5.0. > > This will be the first NumPy release to include support for Python 3, as > > well as for Python 2.7. Please try this beta and report any problems on > the > > NumPy mailing list. > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > Please note that binaries for Python 3.1 are not yet up, they will follow > as > > soon as a minor issue with building them is resolved. Building from > source > > with Python 3.1 should work without problems. > > I gave it a run on the Debian packaging system and these are the results: > > - python 3.1 can't compile it: > > $ python3 setup.py build > Traceback (most recent call last): > File "setup.py", line 210, in > setup_package() > File "setup.py", line 174, in setup_package > import py3tool > ImportError: No module named py3tool > > but this is already known, py3tool is missing from the tarball. > > - python2.6 build, installation and numpy.test() works fine > > - I have a problem building documentation: > > # build doc only for default python version > (export MPLCONFIGDIR=. ; make -C doc html > PYTHONPATH=../build/lib.linux-x86_64-2.6) > make[1]: Entering directory `/tmp/buildd/python-numpy-1.5.0~b1/doc' > mkdir -p build > python \ > ./sphinxext/autosummary_generate.py source/reference/*.rst \ > -p dump.xml -o source/reference/generated > ./sphinxext/autosummary_generate.py:18: DeprecationWarning: The > numpydoc.autosummary extension can also be found as > sphinx.ext.autosummary in Sphinx >= 0.6, and the version in Sphinx >= > 0.7 is superior to the one in numpydoc. This numpydoc version of > autosummary is no longer maintained. > from autosummary import import_by_name > Failed to import 'numpy.__array_priority__': > Failed to import 'numpy.core.defchararray.len': > Failed to import 'numpy.generic.__squeeze__': > touch build/generate-stamp > mkdir -p build/html build/doctrees > LANG=C sphinx-build -b html -d build/doctrees source build/html > Running Sphinx v0.6.6 > > Extension error: > Could not import extension numpydoc (exception: No module named domains.c) > 1.5b1 1.5.0b1 > That's because of the version of Sphinx, domains were introduced in version 1.0. With that you should be able to build the docs. Cheers, Ralf > > I don't know exactly the reason for the "Failed to import X" but they > are there also for 1.4.1 so they should not be a problem. Anyhow, > there is no file named 'domains.c' in the tarball. > > Regards, > -- > Sandro Tosi (aka morph, morpheus, matrixhasu) > My website: http://matrixhasu.altervista.org/ > Me at Debian: http://wiki.debian.org/SandroTosi > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Mon Aug 16 11:51:50 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 16 Aug 2010 11:51:50 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: On Mon, Aug 16, 2010 at 11:36 AM, Ralf Gommers wrote: > > > On Mon, Aug 16, 2010 at 4:50 PM, Sandro Tosi wrote: >> >> Hi all, >> sorry for the delay >> >> On Sun, Aug 1, 2010 at 18:38, Ralf Gommers >> wrote: >> > I am pleased to announce the availability of the first beta of NumPy >> > 1.5.0. >> > This will be the first NumPy release to include support for Python 3, as >> > well as for Python 2.7. Please try this beta and report any problems on >> > the >> > NumPy mailing list. >> > >> > Binaries, sources and release notes can be found at >> > https://sourceforge.net/projects/numpy/files/ >> > Please note that binaries for Python 3.1 are not yet up, they will >> > follow as >> > soon as a minor issue with building them is resolved. Building from >> > source >> > with Python 3.1 should work without problems. >> >> I gave it a run on the Debian packaging system and these are the results: >> >> - python 3.1 can't compile it: >> >> $ python3 setup.py build >> Traceback (most recent call last): >> ?File "setup.py", line 210, in >> ? ?setup_package() >> ?File "setup.py", line 174, in setup_package >> ? ?import py3tool >> ImportError: No module named py3tool >> >> but this is already known, py3tool is missing from the tarball. >> >> - python2.6 build, installation and numpy.test() works fine >> >> - I have a problem building documentation: >> >> # build doc only for default python version >> (export MPLCONFIGDIR=. ; make -C doc html >> PYTHONPATH=../build/lib.linux-x86_64-2.6) >> make[1]: Entering directory `/tmp/buildd/python-numpy-1.5.0~b1/doc' >> mkdir -p build >> python \ >> ? ? ? ? ? ? ? ?./sphinxext/autosummary_generate.py source/reference/*.rst >> \ >> ? ? ? ? ? ? ? ?-p dump.xml -o source/reference/generated >> ./sphinxext/autosummary_generate.py:18: DeprecationWarning: The >> numpydoc.autosummary extension can also be found as >> sphinx.ext.autosummary in Sphinx >= 0.6, and the version in Sphinx >= >> 0.7 is superior to the one in numpydoc. This numpydoc version of >> autosummary is no longer maintained. >> ?from autosummary import import_by_name >> Failed to import 'numpy.__array_priority__': >> Failed to import 'numpy.core.defchararray.len': >> Failed to import 'numpy.generic.__squeeze__': >> touch build/generate-stamp >> mkdir -p build/html build/doctrees >> LANG=C sphinx-build -b html -d build/doctrees ? source build/html >> Running Sphinx v0.6.6 >> >> Extension error: >> Could not import extension numpydoc (exception: No module named domains.c) >> 1.5b1 1.5.0b1 > > That's because of the version of Sphinx, domains were introduced in version > 1.0. With that you should be able to build the docs. > Ah, glad that the directives bug got fixed. I get the same error as above with Sphinx 1.0. Different error with Sphinx 1.0.2. In [2]: np.__version__ Out[2]: '2.0.0.dev8645' $ make html mkdir -p build touch build/generate-stamp mkdir -p build/html build/doctrees LANG=C sphinx-build -b html -d build/doctrees source build/html Running Sphinx v1.0.2 2.0.dev8645 2.0.0.dev8645 Exception occurred: File "/home/skipper/numpy/doc/sphinxext/docscrape.py", line 208, in parse_item_name raise ValueError("%s is not a item name" % text) ValueError: is not a item name The full traceback has been saved in /tmp/sphinx-err-eHqxHV.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. Either send bugs to the mailing list at , or report them in the tracker at . Thanks! make: *** [html] Error 1 Full output: http://pastebin.com/3yGHa1wu Log output: http://pastebin.com/JLbVHjLY Also upgraded to docutils 0.7 to be sure, and the error persists. Any ideas? Skipper From efiring at hawaii.edu Mon Aug 16 12:53:15 2010 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 16 Aug 2010 06:53:15 -1000 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> Message-ID: <4C696CFB.6010808@hawaii.edu> On 08/15/2010 11:14 PM, Sturla Molden wrote: > Two criticial import libraries for mingw-w64 are missing (libpython26.a > and msvcr90.a). We cannot build C extensions for Python with mingw-w64. If > you can build with disutils, your are not using mingw-w64 but another C > compiler. Sturla, Have you tried the Equation Solution versions of the gcc tools? http://www.equation.com/servlet/equation.cmd?fa=fortran Eric > > There are rumors of incompatibility issues between libgfortran and the CRT > or SciPy on Windows 64. David Cournapeau might know the details. > > It seems f2py on Windows 64 requires a commercial Fortran compiler (the > ones worth looking at are Absoft, Intel and PGI) and e.g. Microsoft's free > C/C++ compiler (download Windows 7 SDK for .NET 3.5 -- not any later or > earlier version). You need to have the environment variables > DISTUTILS_USE_SDK and MSSdk set to build with the free SDK compiler. f2py > will enforce the former, the latter comes from running setenv from the > SDK. The latter is important as you will need to set up the environments > for the Fortran and the C compiler in the same command window before > running f2py. > > I also had to remove support for Compaq Visual Fortran from the f2py > source to make anything build, as it crashes f2py on Windows 64 (this > does not happen on 32-bit Windows). > > Building f2py extensions on Windows 64 is a bit more tricky. And I've had > no luck with gfortran so far. > > Sturla > > > >> Hi all, >> >> Are there any success stories in building f2py extensions on AMD64 with >> latest versions? Building the same extension on 32 bit works like a charm. >> >> I am having trouble finding documentation or examples, is it supposed to >> be >> working? >> >> Compiling (with distutils) works like a charm, but that does not help when >> I'm stuck figuring out link dependencies. It seems to me that the gfortran >> library depends on a mingw that is in conflict with some CRT library. >> >> I have to admit, I'm probably in way too deep waters, and should really be >> satisfied with 32 bit, but still, it would be fun to get it working. >> >> GFortran is from MinGW-w64 project on sourceforge, version 4.5.1 >> prerelease. >> >> Any pointers or other experiences? >> >> Regards, >> ?smund Hjulstad >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From cournape at gmail.com Mon Aug 16 13:02:08 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 17 Aug 2010 02:02:08 +0900 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: <4C696CFB.6010808@hawaii.edu> References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> <4C696CFB.6010808@hawaii.edu> Message-ID: On Tue, Aug 17, 2010 at 1:53 AM, Eric Firing wrote: > On 08/15/2010 11:14 PM, Sturla Molden wrote: >> Two criticial import libraries for mingw-w64 are missing (libpython26.a >> and msvcr90.a). We cannot build C extensions for Python with mingw-w64. If >> you can build with disutils, your are not using mingw-w64 but another C >> compiler. > > Sturla, > > Have you tried the Equation Solution versions of the gcc tools? > http://www.equation.com/servlet/equation.cmd?fa=fortran It has the same issues last time I checked. The problem is that libgfortran relies a lot on the mingw runtime, which has incompatibilities with the msvc C runtime - much more it seems that the g77 ever did. The fact that it worked for scipy was pure luck, though. I think the right solution is to build our own libgfortran, but against the MSVC C runtime instead of the mingw runtime. THis is not as crazy as it sounds because scipy does not rely so much on the fortran runtime. Just faking the library (by putting functions which do nothing), I could run half of scipy last time I tried. I also started working on removing most of the fortran IO code in scipy (which would cause issues anyway independently of this issue), but then I started my new job :) It is unlikely I will work on this again in the near future, but it should not be so hard for anyone who has incentive to make this work, cheers, David > Eric > > >> >> There are rumors of incompatibility issues between libgfortran and the CRT >> or SciPy on Windows 64. David Cournapeau might know the details. >> >> It seems f2py on Windows 64 requires a commercial Fortran compiler (the >> ones worth looking at are Absoft, Intel and PGI) and e.g. Microsoft's free >> C/C++ compiler (download Windows 7 SDK for .NET 3.5 -- not any later or >> earlier version). You need to have the environment variables >> DISTUTILS_USE_SDK and MSSdk set to build with the free SDK compiler. f2py >> will enforce the former, the latter comes from running setenv from the >> SDK. The latter is important as you will need to set up the environments >> for the Fortran and the C compiler in the same command window before >> running f2py. >> >> I also had to remove support for Compaq Visual Fortran from the f2py >> source to ?make anything build, as it crashes f2py on Windows 64 (this >> does not happen on 32-bit Windows). >> >> Building f2py extensions on Windows 64 is a bit more tricky. And I've had >> no luck with gfortran so far. >> >> Sturla >> >> >> >>> Hi all, >>> >>> Are there any success stories in building f2py extensions on AMD64 with >>> latest versions? Building the same extension on 32 bit works like a charm. >>> >>> I am having trouble finding documentation or examples, is it supposed to >>> be >>> working? >>> >>> Compiling (with distutils) works like a charm, but that does not help when >>> I'm stuck figuring out link dependencies. It seems to me that the gfortran >>> library depends on a mingw that is in conflict with some CRT library. >>> >>> I have to admit, I'm probably in way too deep waters, and should really be >>> satisfied with 32 bit, but still, it would be fun to get it working. >>> >>> GFortran is from ?MinGW-w64 project on sourceforge, version 4.5.1 >>> prerelease. >>> >>> Any pointers or other experiences? >>> >>> Regards, >>> ?smund Hjulstad >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sturla at molden.no Mon Aug 16 13:24:09 2010 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Aug 2010 19:24:09 +0200 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> <4C696CFB.6010808@hawaii.edu> Message-ID: <0d970e09d7025f997005ce4400187e6a.squirrel@webmail.uio.no> Thank for the information, David. > I think the right solution is to build our own libgfortran, but > against the MSVC C runtime instead of the mingw runtime. THis is not > as crazy as it sounds because scipy does not rely so much on the > fortran runtime. Just faking the library (by putting functions which > do nothing), That might work for building SciPy, but not for arbitrary Fortran 95 code. So the options are to use another compiler or run gfortran code in a spawned subprocess. Sturla From sturla at molden.no Mon Aug 16 13:36:01 2010 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Aug 2010 19:36:01 +0200 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: <4C696CFB.6010808@hawaii.edu> References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> <4C696CFB.6010808@hawaii.edu> Message-ID: <9d1902179a3e3984a979f9bb0d0b1e43.squirrel@webmail.uio.no> > Sturla, > > Have you tried the Equation Solution versions of the gcc tools? > http://www.equation.com/servlet/equation.cmd?fa=fortran > > Eric It does not matter which mingw-w64 build we use. The import libraries will still be missing, and the issies with libgfortran are still there. Also, AVG and McAfee warns about a trojan in the installers from equation.com. I am not taking any chances with a compromised C compiler. Imagine a compiler that infects anything you build with malware. That is about as bad as it gets. I'd rather play safe and use a mingw build from a trusted source; one that antivirus software don't warn me about. Building mingw-w64 from gcc sources is not rocket science either. Personally I use Absoft Pro Fortran and the MSVC compiler. Sturla From gberbeglia at gmail.com Mon Aug 16 14:25:00 2010 From: gberbeglia at gmail.com (gerardob) Date: Mon, 16 Aug 2010 11:25:00 -0700 (PDT) Subject: [Numpy-discussion] basic question about numpy arrays Message-ID: <29449184.post@talk.nabble.com> I have a numpy array A , such that when i print A it appears: [[ 10.] [ 20.]] I would like to have a one dimensional array B (obtained from A) such that B[0] = 10 and B[1]=20. It could be seen as the transpose of A. How can i obtain B = [10,20] from A? I tried transpose(1,0) but it doesn't seem to be useful. Thanks. -- View this message in context: http://old.nabble.com/basic-question-about-numpy-arrays-tp29449184p29449184.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From jsseabold at gmail.com Mon Aug 16 14:28:03 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 16 Aug 2010 14:28:03 -0400 Subject: [Numpy-discussion] basic question about numpy arrays In-Reply-To: <29449184.post@talk.nabble.com> References: <29449184.post@talk.nabble.com> Message-ID: On Mon, Aug 16, 2010 at 2:25 PM, gerardob wrote: > > I have a numpy array A , such that when i print A it appears: > > [[ 10.] > ?[ 20.]] > > I would like to have a one dimensional array B (obtained from A) such that > B[0] = 10 and B[1]=20. It could be seen as the transpose of A. > > How can i obtain B = [10,20] ?from A? I tried transpose(1,0) but it doesn't > seem to be useful. > B = A.squeeze() or B = A.flatten() would be two ways. Skipper From jsalvati at u.washington.edu Mon Aug 16 14:30:35 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 16 Aug 2010 11:30:35 -0700 Subject: [Numpy-discussion] basic question about numpy arrays In-Reply-To: References: <29449184.post@talk.nabble.com> Message-ID: You can also do: y = x[:,0] On Mon, Aug 16, 2010 at 11:28 AM, Skipper Seabold wrote: > On Mon, Aug 16, 2010 at 2:25 PM, gerardob wrote: > > > > I have a numpy array A , such that when i print A it appears: > > > > [[ 10.] > > [ 20.]] > > > > I would like to have a one dimensional array B (obtained from A) such > that > > B[0] = 10 and B[1]=20. It could be seen as the transpose of A. > > > > How can i obtain B = [10,20] from A? I tried transpose(1,0) but it > doesn't > > seem to be useful. > > > > B = A.squeeze() > > or > > B = A.flatten() > > would be two ways. > > Skipper > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rspeer at MIT.EDU Mon Aug 16 14:35:36 2010 From: rspeer at MIT.EDU (Rob Speer) Date: Mon, 16 Aug 2010 14:35:36 -0400 Subject: [Numpy-discussion] basic question about numpy arrays In-Reply-To: <29449184.post@talk.nabble.com> References: <29449184.post@talk.nabble.com> Message-ID: To explain: A has shape (2,1), meaning it's a 2-D array with 2 rows and 1 column. The transpose of A has shape (1,2): it's a 2-D array with 1 row and 2 columns. That's not the same as what you want, which is an array with shape (2,): a 1-D array with 2 entries. When you take A[:,0], you're pulling out the 1-D array that constitutes column 0 of your array, which is exactly what you want. -- Rob From efiring at hawaii.edu Mon Aug 16 14:56:40 2010 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 16 Aug 2010 08:56:40 -1000 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: <9d1902179a3e3984a979f9bb0d0b1e43.squirrel@webmail.uio.no> References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> <4C696CFB.6010808@hawaii.edu> <9d1902179a3e3984a979f9bb0d0b1e43.squirrel@webmail.uio.no> Message-ID: <4C6989E8.5050703@hawaii.edu> On 08/16/2010 07:36 AM, Sturla Molden wrote: > >> Sturla, >> >> Have you tried the Equation Solution versions of the gcc tools? >> http://www.equation.com/servlet/equation.cmd?fa=fortran >> >> Eric > > > It does not matter which mingw-w64 build we use. The import libraries will > still be missing, and the issies with libgfortran are still there. Also, > AVG and McAfee warns about a trojan in the installers from equation.com. I Here is the rebuttal from Equation Solution: http://www.equation.com/servlet/equation.cmd?fa=falsevirus By the way, thank you for working on python-dev toward a solution to the problem of building extensions for Win using free compilers. (http://www.gossamer-threads.com/lists/python/dev/854392). Eric > am not taking any chances with a compromised C compiler. Imagine a > compiler that infects anything you build with malware. That is about as > bad as it gets. I'd rather play safe and use a mingw build from a trusted > source; one that antivirus software don't warn me about. Building > mingw-w64 from gcc sources is not rocket science either. > > Personally I use Absoft Pro Fortran and the MSVC compiler. > > Sturla From v.gkinis at nbi.ku.dk Mon Aug 16 12:34:34 2010 From: v.gkinis at nbi.ku.dk (Vasileios Gkinis) Date: Mon, 16 Aug 2010 16:34:34 +0000 (UTC) Subject: [Numpy-discussion] f2py performance question from a rookie References: <4C690106.80407@nbi.ku.dk> Message-ID: Sturla Molden molden.no> writes: > > > > It looks like the gain in performance is > > rather low compared to tests i have seen elsewhere.
> >
> > Am I missing something here..?
> >
> > Cheers...Vasilis
> > Turn HTML off please. > > Use time.clock(), not time.time(). > > Try some tasks that actually takes a while. Tasks that take 10**-4 or > 10**-3 seconds cannot be reliably timed on Windows or Linux. > Hi again, After using time.clock, running f2py with the REPORT_ON_ARRAY_COPY enabled and passing arrays as np.asfortranarray(array) to the fortran routines I still get a slow performance on f2py. No copied arrays are reported. Running on f2py with a 6499x6499 array takes about 1.2sec while the python for loop does the job slightly faster at 0.9 sec. Comparisons like this: http://www.scipy.org/PerformancePython indicate a 100x-1000x boost in performance with f2py when compared to conventional python for loops. Still quite puzzled...any help will be very much appreciated Regards Vasileios The actual fortran subroutine is here: subroutine fill_B(j, beta, lamda, mu, b) integer, intent(in) :: j integer :: row, col real(kind = 8), intent(in) :: beta(j), lamda(j), mu(j) real(kind = 8), intent(out) :: b(j,j) do row=2,j-1 do col=1,j if (col == row-1) then b(row,col) = beta(row) - lamda(row) + mu(row) elseif (col == row) then b(row,col) = 1 - 2*beta(row) elseif (col == row+1) then b(row, col) = beta(row) + lamda(row) - mu(row) else b(row, col) = 0 endif enddo enddo b(1,1) = 1 b(j,j) = 1 end and the python code that calls it together with the alternative implementation with conventional python for loops is here: def crank_heat(self, profile_i, go, g1, beta, lamda, mu, usef2py = False): """ """ ##Crank Nicolson AX = BD ##Make matrix A (coefs of n+1 step N = np.size(profile_i) print N t1 = time.clock() if usef2py == True: matrix_A = fill_A.fill_a(j = N, beta = beta, lamda = lamda, mu = mu) matrix_B = fill_B.fill_b(j = N, beta = beta, lamda = lamda, mu = mu) else: matrix_A = np.zeros((N,N)) matrix_A[0,0] = 1 matrix_A[-1,-1] = 1 for row in np.arange(1,N-1): matrix_A[row,row-1] = -(beta[row] - lamda[row] + mu[row]) matrix_A[row, row] = 1 + 2*beta[row] matrix_A[row, row+1] = -(beta[row] + lamda[row] - mu[row]) #make matrix B matrix_B = np.zeros((N,N)) matrix_B[0,0] = 1 matrix_B[-1,-1] = 1 for row in np.arange(1,N-1): matrix_B[row,row-1] = beta[row] - lamda[row] + mu[row] matrix_B[row, row] = 1 - 2*beta[row] matrix_B[row, row+1] = beta[row] + lamda[row] - mu[row] print("CN function time: %0.5e" %(time.clock()-t1)) matrix_C = np.dot(matrix_B, profile_i) t1 = time.clock() matrix_X = np.linalg.solve(matrix_A, matrix_C) print("CN solve time: %0.5e" %(time.clock()-t1)) matrix_X[0] = go matrix_X[-1] = g1 return matrix_X From sturla at molden.no Mon Aug 16 18:01:29 2010 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Aug 2010 00:01:29 +0200 Subject: [Numpy-discussion] f2py performance question from a rookie In-Reply-To: References: <4C690106.80407@nbi.ku.dk> Message-ID: > After using time.clock, running f2py with the REPORT_ON_ARRAY_COPY enabled > and > passing arrays as np.asfortranarray(array) to the fortran routines I still > get a > slow performance on f2py. No copied arrays are reported. That is not any better as np.asfortranarray will make a copy instead. Pass the transpose to Fortran, and transpose the return value. Make sure you neither make a copy before or after calling Fortran. A transpose does not make a copy in NumPy. You want C ordering in NumPy and Fortran ordering in Fortran. Which means you probably want to call Fortran like this: foobar(x.T, y.T) If you let f2py return y as a result, make sure it is C ordered after transpose: y = foobar(x.T).T assert(y.flags['C_CONTIGUOUS']) Even if you don't get a "copy and transpose" immediately (i.e. f2py does not report a copy), it might happen silently later on if you get Fortran ordered arrays returned into Python. > Running on f2py with a 6499x6499 array takes about 1.2sec while the python > for > loop does the job slightly faster at 0.9 sec. I suppose you are including the solve time here? In which case the dominating factors are np.dot and np.linalg.solve. Fortran will not help a lot then. Well if you have access to performance libraries (ACML or MKL) you might save a little by using Fortran DGEMM instead of np.dot and a carefully selected Fortran LAPACK solver (that is, np.linalg.solve use Gaussian substitution, i.e. LU, but there might be structure in your matrices to exploit, I haven't looked too carefully.) > subroutine fill_B(j, beta, lamda, mu, b) > integer, intent(in) :: j > integer :: row, col > real(kind = 8), intent(in) :: beta(j), lamda(j), mu(j) > real(kind = 8), intent(out) :: b(j,j) > > do row=2,j-1 > do col=1,j > if (col == row-1) then > b(row,col) = beta(row) - lamda(row) + mu(row) > elseif (col == row) then > b(row,col) = 1 - 2*beta(row) > elseif (col == row+1) then > b(row, col) = beta(row) + lamda(row) - mu(row) > else > b(row, col) = 0 > endif > enddo > enddo > b(1,1) = 1 > b(j,j) = 1 > > end That iterates over strided memory in Fortran. Memory access is slow, computation is fast. So there are two problems with your Fortran code: 1. You want b to be transposed in Fortran compared to Python. 2. Iterate in column major order in Fortran, not row major order. Sturla From cournape at gmail.com Mon Aug 16 18:08:17 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 17 Aug 2010 07:08:17 +0900 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: <0d970e09d7025f997005ce4400187e6a.squirrel@webmail.uio.no> References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> <4C696CFB.6010808@hawaii.edu> <0d970e09d7025f997005ce4400187e6a.squirrel@webmail.uio.no> Message-ID: On Tue, Aug 17, 2010 at 2:24 AM, Sturla Molden wrote: > > Thank for the information, David. > >> I think the right solution is to build our own libgfortran, but >> against the MSVC C runtime instead of the mingw runtime. THis is not >> as crazy as it sounds because scipy does not rely so much on the >> fortran runtime. Just faking the library (by putting functions which >> do nothing), > > That might work for building SciPy, but not for arbitrary Fortran 95 code. Actually, it does, you just have to build the whole libgfortran to support arbitrary code. > So the options are to use another compiler or run gfortran code in a > spawned subprocess. I don't really see how to make that works (that is running gfortran code in a spawned process), but if you know how to, please go ahead. David From sturla at molden.no Mon Aug 16 21:59:30 2010 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Aug 2010 03:59:30 +0200 Subject: [Numpy-discussion] Building f2py extensions on AMD64 with Python 2.7 Numpy 1.5, MSVC 2008 Pro and gfortran In-Reply-To: References: <49398f08d9aa27e813b66fca2dfc0c03.squirrel@webmail.uio.no> <4C696CFB.6010808@hawaii.edu> <0d970e09d7025f997005ce4400187e6a.squirrel@webmail.uio.no> Message-ID: <70e09e3cfcbe82fb58a0f2cdca411386.squirrel@webmail.uio.no> > I don't really see how to make that works (that is running gfortran > code in a spawned process), but if you know how to, please go ahead. I didn't debug this code examples so there might be typos, but this is just to illustrate the general idea: Say we have a Fortran function with this interface: subroutine foobar(m, n, x, y) integer :: m, n real*8 :: x(n), y(n) end subroutine We could then proceed like this: We can create a chunk of uniquely named shared memory like this: tag = uuid.uuid4().hex size = 1 << 20 # 1 MiB from the paging file shm = mmap.mmap(0, size, tagname="Local\\%s"%(tag,)) shm = np.frombuffer(shm, dtype=np.uint8) The we use views of shm to set up x and y, e.g. m = 100 n = 100 nb = 8 * n * m x = shm[:nb].view(dtype=float).reshape((n,m)) y = shm[nb:2*nb].view(dtype=float).reshape((n,m)) Then we can call a fortran program using subprocess module: returnval = subprocess.call(['foobar.exe','%d'%n, '%d'%m, tag]) Then we need some boilerplate C to grab the shared memory and call Fortran: #include #include #include /* fortran: pass pointers, append underscore */ extern void foobar_(int *m, int *n, double *x, double *y) static int openshm(const char *tag, size_t size, HANDLE *phandle, void **pbuf) { *phandle = OpenFileMapping(FILE_MAP_ALL_ACCESS, TRUE, tag); if (!h) return -1; *pbuf = MapViewOfFile(*phandle, FILE_MAP_ALL_ACCESS, 0, 0, size); if (*pbuf == NULL) { CloseHandle(p); return -1; } return 0; } int main(int argc, char *argv[]) { char tag[256]; HANDLE shmem; int m, n; double *x, *y; m = atoi(argv[1]); n = atoi(argv[2]); ZeroMemory(tag, sizeof(tag)); sprintf(tag, "Local\\%s", argv[3]); /* get arrays from shared memory */ if (openshm(tag, 2*m*n*sizeof(double), &shmem, &(void*)x)<0) return -1; y = x + m*n; /* call fortran */ foobar_(&m, &n, x, y); /* done */ CloseHandle(h); return 0; } Blah... ugly. On Linux we could use os.fork instead of subprocess, and only y would need to be placed in (anonymous) shared memory (due to copy-on-write). But on the other hand, we don't have run-time issues with gfortran on Linux either. Just to illustrate why Windows sucks sometimes. :-) Sturla From charlesr.harris at gmail.com Mon Aug 16 23:00:13 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Aug 2010 21:00:13 -0600 Subject: [Numpy-discussion] Seeking advice on crowded namespace. Message-ID: Hi All, I just added support for Legendre polynomials to numpy and I think the numpy.polynomial name space is getting a bit crowded. Since most of the current functions in that namespace are just used to implement the Polynomial, Chebyshev, and Legendre classes I'm thinking of only importing those classes by default and leaving the other functions to explicit imports. Of course I will have to fix the examples and maybe some other users will be inconvenienced by the change. But with 2.0.0 in the works this might be a good time to do this. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ater1980 at gmail.com Tue Aug 17 00:13:07 2010 From: ater1980 at gmail.com (Alex Ter-Sarkissov) Date: Tue, 17 Aug 2010 16:13:07 +1200 Subject: [Numpy-discussion] array manipulation Message-ID: hi, this is probably a very silly question, but I can't get my hear around it unfortunately( I have an array (say, mat=rand(3,5)) from which I 'pull out' a row (say, s1=mat[1,]). The problem is, the shape of this row s1 is not [1,5], as I would expect, but rather [5,], which means that I can't, for example, concateante mat and s1 rowwise. thanks for the help -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Aug 17 00:25:00 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Aug 2010 00:25:00 -0400 Subject: [Numpy-discussion] array manipulation In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 12:13 AM, Alex Ter-Sarkissov wrote: > hi, this is probably a very silly question, but I can't get my hear > around it unfortunately( > > I have an array (say, mat=rand(3,5)) from which I 'pull out' a row > (say, s1=mat[1,]). The problem is, the shape of this row s1 is not > [1,5], as I would expect, but rather [5,], which means that I can't, > for example, concateante mat and s1 rowwise. try np.vstack((mat[0], mat)) np.arrays are not matrices, a slice with a single number reduces dimenson, mat[0] which is identical to mat[0,:] for 2d array mat[0][np.newaxis,:] adds a new (row) dimension >>> mat[0][np.newaxis,:].shape (1, 5) it is described somewhere in the help on slicing and indexing Josef > > thanks for the help > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ashford at whisperpc.com Tue Aug 17 00:58:09 2010 From: ashford at whisperpc.com (ashford at whisperpc.com) Date: Mon, 16 Aug 2010 21:58:09 -0700 (PDT) Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS Message-ID: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> I'm having problems getting the GotoBLAS library (Nehalem optimized BLAS - "http://www.tacc.utexas.edu/tacc-projects/gotoblas2/") working properly under the Python NumPy package ("http://numpy.scipy.org/") on a quad-core Nehalem under FC10. The command used to build the library is: make BINARY=64 USE_THREAD=1 MAX_CPU_NUMBER=4 I'm limiting this to four cores, as I believe HyperThreading will slow it down (I've seen this happen with other scientific code). I'll benchmark later to see whether or not HyperThreading helps. I built the library (it uses -fPIC), then installed it under /usr/local/lib64, and created the appropriate links: # cp libgoto2_nehalemp-r1.13.a /usr/local/lib64 # cp libgoto2_nehalemp-r1.13.so /usr/local/lib64 # cd /usr/local/lib64 # ln -s libgoto2_nehalemp-r1.13.a libgoto2.a # ln -s libgoto2_nehalemp-r1.13.so libgoto2.so # ln -s libgoto2_nehalemp-r1.13.a libblas.a # ln -s libgoto2_nehalemp-r1.13.so libblas.so Without the libblas links, the NumPy configuration used the system default BLAS library (single-threaded NetLib under FC10); it's set up for the NetLib and Atlas BLAS libraries. I used the latest release of NumPy, with no site.cfg file, and no NumPy directory ("rm -rf /usr/local/python2.6/lib/python2.6/site-packages/numpy"). The configuration step ("python setup.py config") appears to run OK, as do the build ("python setup.py build") and install ("python setup.py install") steps. The problem is that the benchmark takes 8.5 seconds, which is what it took before I changed the library. python -c "import numpy as N; a=N.random.randn(1000, 1000); N.dot(a, a)" I expect I'm missing something really simple here, but I've spent >10 hours on it, and I have no idea as to what it could be. I've tried various permutations on the site.cfg file, all to no avail. I've also tried different names on the library, and different locations. I've even tried a set of symbolic links in /usr/local/lib64 for liblapack that point to libgoto. Could someone offer some suggestions? Thank you for your time. Peter Ashford From david at silveregg.co.jp Tue Aug 17 01:09:38 2010 From: david at silveregg.co.jp (David) Date: Tue, 17 Aug 2010 14:09:38 +0900 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> Message-ID: <4C6A1992.6050608@silveregg.co.jp> On 08/17/2010 01:58 PM, ashford at whisperpc.com wrote: > I'm having problems getting the GotoBLAS library (Nehalem optimized BLAS - > "http://www.tacc.utexas.edu/tacc-projects/gotoblas2/") working properly under > the Python NumPy package ("http://numpy.scipy.org/") on a quad-core Nehalem > under FC10. > > The command used to build the library is: > make BINARY=64 USE_THREAD=1 MAX_CPU_NUMBER=4 > > I'm limiting this to four cores, as I believe HyperThreading will slow it down > (I've seen this happen with other scientific code). I'll benchmark later to > see whether or not HyperThreading helps. > > I built the library (it uses -fPIC), then installed it under /usr/local/lib64, > and created the appropriate links: > # cp libgoto2_nehalemp-r1.13.a /usr/local/lib64 > # cp libgoto2_nehalemp-r1.13.so /usr/local/lib64 > # cd /usr/local/lib64 > # ln -s libgoto2_nehalemp-r1.13.a libgoto2.a > # ln -s libgoto2_nehalemp-r1.13.so libgoto2.so > # ln -s libgoto2_nehalemp-r1.13.a libblas.a > # ln -s libgoto2_nehalemp-r1.13.so libblas.so The .so are only used when linking, and not the ones used at runtime generally (the full version, e.g. .so.1.2.3 is). Which version exactly depends on your installation, but I actually advise you against doing those softlink. You should instead specificaly link the GOTO library to numpy, by customizing the site.cfg, cheers, David From ashford at whisperpc.com Tue Aug 17 01:14:29 2010 From: ashford at whisperpc.com (ashford at whisperpc.com) Date: Mon, 16 Aug 2010 22:14:29 -0700 (PDT) Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <4C6A1992.6050608@silveregg.co.jp> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <4C6A1992.6050608@silveregg.co.jp> Message-ID: <33559.66.146.167.66.1282022069.squirrel@phoenix.liquidweb.com> David, > You should instead specificaly link the GOTO library to > numpy, by customizing the site.cfg, That was one of the many things I tried came to the list. Peter Ashford From fperez.net at gmail.com Tue Aug 17 02:07:51 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Aug 2010 23:07:51 -0700 Subject: [Numpy-discussion] py3k execfile('setup.py') In-Reply-To: References: Message-ID: On Sat, Aug 14, 2010 at 6:11 AM, Ralf Gommers wrote: > Installing into a local virtualenv (with either of the methods that work) > looks good. Into /Library/Frameworks/..., nose refuses to run any tests, due > to the files being executable. But I think that's normal. Changing > permissions to 644 manually it all looks good again. You can also pass --exe to nose to override this safety, if you want (output of nosetests --help below): --exe Look for tests in python modules that are executable. Normal behavior is to exclude executable modules, since they may not be import-safe [NOSE_INCLUDE_EXE] --noexe DO NOT look for tests in python modules that are executable. (The default on the windows platform is to do so.) Cheers, f ps - congrats to all of you for the progress on the py3k work! From alan.isaac at gmail.com Tue Aug 17 06:50:54 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 17 Aug 2010 06:50:54 -0400 Subject: [Numpy-discussion] array manipulation In-Reply-To: References: Message-ID: <4C6A698E.5090602@gmail.com> On 8/17/2010 12:13 AM, Alex Ter-Sarkissov wrote: > I have an array (say, mat=rand(3,5)) from which I 'pull out' a row > (say, s1=mat[1,]). The problem is, the shape of this row s1 is not > [1,5], as I would expect, but rather [5,], which means that I can't, > for example, concateante mat and s1 rowwise. Your ``mat`` is an array. You get the behavior you seek with NumPy matrices (although I personally consider is a misfeature). >>> m = np.mat(mat) >>> m[1] matrix([[ 0.95488909, 0.55174423, 0.12103472, 0.15482402, 0.71286625]]) hth, Alan Isaac From ralf.gommers at googlemail.com Tue Aug 17 07:36:59 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 17 Aug 2010 19:36:59 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 1 In-Reply-To: References: Message-ID: On Mon, Aug 16, 2010 at 11:51 PM, Skipper Seabold wrote: > On Mon, Aug 16, 2010 at 11:36 AM, Ralf Gommers > wrote: > > > > > > On Mon, Aug 16, 2010 at 4:50 PM, Sandro Tosi wrote: > >> > >> Hi all, > >> sorry for the delay > >> > >> On Sun, Aug 1, 2010 at 18:38, Ralf Gommers > > >> wrote: > >> > I am pleased to announce the availability of the first beta of NumPy > >> > 1.5.0. > >> > This will be the first NumPy release to include support for Python 3, > as > >> > well as for Python 2.7. Please try this beta and report any problems > on > >> > the > >> > NumPy mailing list. > >> > > >> > Binaries, sources and release notes can be found at > >> > https://sourceforge.net/projects/numpy/files/ > >> > Please note that binaries for Python 3.1 are not yet up, they will > >> > follow as > >> > soon as a minor issue with building them is resolved. Building from > >> > source > >> > with Python 3.1 should work without problems. > >> > >> I gave it a run on the Debian packaging system and these are the > results: > >> > >> - python 3.1 can't compile it: > >> > >> $ python3 setup.py build > >> Traceback (most recent call last): > >> File "setup.py", line 210, in > >> setup_package() > >> File "setup.py", line 174, in setup_package > >> import py3tool > >> ImportError: No module named py3tool > >> > >> but this is already known, py3tool is missing from the tarball. > >> > >> - python2.6 build, installation and numpy.test() works fine > >> > >> - I have a problem building documentation: > >> > >> # build doc only for default python version > >> (export MPLCONFIGDIR=. ; make -C doc html > >> PYTHONPATH=../build/lib.linux-x86_64-2.6) > >> make[1]: Entering directory `/tmp/buildd/python-numpy-1.5.0~b1/doc' > >> mkdir -p build > >> python \ > >> ./sphinxext/autosummary_generate.py > source/reference/*.rst > >> \ > >> -p dump.xml -o source/reference/generated > >> ./sphinxext/autosummary_generate.py:18: DeprecationWarning: The > >> numpydoc.autosummary extension can also be found as > >> sphinx.ext.autosummary in Sphinx >= 0.6, and the version in Sphinx >= > >> 0.7 is superior to the one in numpydoc. This numpydoc version of > >> autosummary is no longer maintained. > >> from autosummary import import_by_name > >> Failed to import 'numpy.__array_priority__': > >> Failed to import 'numpy.core.defchararray.len': > >> Failed to import 'numpy.generic.__squeeze__': > >> touch build/generate-stamp > >> mkdir -p build/html build/doctrees > >> LANG=C sphinx-build -b html -d build/doctrees source build/html > >> Running Sphinx v0.6.6 > >> > >> Extension error: > >> Could not import extension numpydoc (exception: No module named > domains.c) > >> 1.5b1 1.5.0b1 > > > > That's because of the version of Sphinx, domains were introduced in > version > > 1.0. With that you should be able to build the docs. > > > > Ah, glad that the directives bug got fixed. I get the same error as > above with Sphinx 1.0. Different error with Sphinx 1.0.2. > > In [2]: np.__version__ > Out[2]: '2.0.0.dev8645' > > $ make html > mkdir -p build > touch build/generate-stamp > mkdir -p build/html build/doctrees > LANG=C sphinx-build -b html -d build/doctrees source build/html > Running Sphinx v1.0.2 > 2.0.dev8645 2.0.0.dev8645 > > > > Exception occurred: > File "/home/skipper/numpy/doc/sphinxext/docscrape.py", line 208, in > parse_item_name > raise ValueError("%s is not a item name" % text) > ValueError: is not a item name > The full traceback has been saved in /tmp/sphinx-err-eHqxHV.log, if > you want to report the issue to the developers. > Please also report this if it was a user error, so that a better error > message can be provided next time. > Either send bugs to the mailing list at > , > or report them in the tracker at > . Thanks! > make: *** [html] Error 1 > > > Full output: http://pastebin.com/3yGHa1wu > Log output: http://pastebin.com/JLbVHjLY > > Also upgraded to docutils 0.7 to be sure, and the error persists. Any > ideas? > I've seen such an error before, but can't remember what the issue was. For the record, my working config is: python 2.6 numpy 1.5.x (can't check trunk right now, I'd have to recompile MPL) sphinx 1.0.1 docutils 0.6 numpydoc 0.3.1 This may be an actual bug in trunk which is not in 1.5.x. It gets quite far reading sources and then chokes on a specific docstring, so I think your toolchain is fine. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From eg at fft.be Tue Aug 17 07:43:43 2010 From: eg at fft.be (Eloi Gaudry) Date: Tue, 17 Aug 2010 13:43:43 +0200 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <33559.66.146.167.66.1282022069.squirrel@phoenix.liquidweb.com> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <4C6A1992.6050608@silveregg.co.jp> <33559.66.146.167.66.1282022069.squirrel@phoenix.liquidweb.com> Message-ID: <201008171343.44255.eg@fft.be> Peter, please below a script that will build numpy using a relevant site.cfg for your configuration (you need to update GOTODIR and LAPACKDIR and PYTHONDIR): #!/bin/sh #BLAS/LAPACK configuration file echo "[blas]" > ./site.cfg echo "library_dirs = GOTODIR" >> ./site.cfg echo "blas_libs = goto2_nehalemp-r1.13" >> ./site.cfg echo "[lapack]" >> ./site.cfg echo "library_dirs = LAPACKDIR" >> ./site.cfg echo "lapack_libs = lapack" >> ./site.cfg #compilation variables export CC=gcc export F77=gfortran export F90=gfortran export F95=gfortran export LDFLAGS="-shared -Wl,-rpath=\'\$ORIGIN/../../../..\'" export CFLAGS="-O1 -pthread" export FFLAGS="-O2" #build python setup.py config python setup.py build python setup.py install #copy site.cfg cp ./site.cfg PYTHONDIR/lib/python2.6/site-packages/numpy/distutils/. # Test PWD=`pwd` cd $HOME python -c "import numpy ; numpy.test()" cd $PWD Regards, Eloi On Tuesday 17 August 2010 07:14:29 ashford at whisperpc.com wrote: > David, > > > You should instead specificaly link the GOTO library to > > numpy, by customizing the site.cfg, > > That was one of the many things I tried came to the list. > > Peter Ashford > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Eloi Gaudry Free Field Technologies Company Website: http://www.fft.be Company Phone: +32 10 487 959 From zbyszek at in.waw.pl Tue Aug 17 09:03:10 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Tue, 17 Aug 2010 15:03:10 +0200 Subject: [Numpy-discussion] numpy.concatenate slower than slice copying Message-ID: <20100817130310.GA12292@in.waw.pl> Hi, this is a problem which came up when trying to replace a hand-written array concatenation with a call to numpy.vstack: for some array sizes, numpy.vstack(data) runs > 20% longer than a loop like alldata = numpy.empty((tlen, dim)) for x in data: step = x.shape[0] alldata[pos:pos+step] = x pos += step (example script attached) $ python del_cum3.py numpy 10000 10000 1 10 problem size: (10000x10000) x 1 = 10^8 0.816s <------------------------------- numpy.concatentate of 10 arrays 10000x10000 $ python del_cum3.py concat 10000 10000 1 10 problem size: (10000x10000) x 1 = 10^8 0.642s <------------------------------- slice manipulation giving the same result When the array size is reduced to 100x100 or so, the computation time goes to 0, so it seems that the dtype and dimension checking is negligible. Does numpy.concatenate do some extra work? Thanks for any pointers, Zbyszek PS. Architecture is amd64. python2.6, numpy 1.3.0 or python3.1, numpy 2.0.0.dev / trunk at 8510 give the same result. -------------- next part -------------- A non-text attachment was scrubbed... Name: del_cum3.py Type: text/x-python Size: 884 bytes Desc: not available URL: From gberbeglia at gmail.com Tue Aug 17 10:11:41 2010 From: gberbeglia at gmail.com (gerardob) Date: Tue, 17 Aug 2010 07:11:41 -0700 (PDT) Subject: [Numpy-discussion] set operation Message-ID: <29449267.post@talk.nabble.com> I would like to create a set composed of the numbers 2,3,4,...49,50,51. How do i do? Thanks. -- View this message in context: http://old.nabble.com/set-operation-tp29449267p29449267.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From ben.root at ou.edu Tue Aug 17 10:30:54 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 17 Aug 2010 09:30:54 -0500 Subject: [Numpy-discussion] set operation In-Reply-To: <29449267.post@talk.nabble.com> References: <29449267.post@talk.nabble.com> Message-ID: On Tue, Aug 17, 2010 at 9:11 AM, gerardob wrote: > > I would like to create a set composed of the numbers 2,3,4,...49,50,51. How > do i do? > Thanks. > Gerard, Well, you can easily create a list of those numbers like so: >>> a = range(2, 52) # note, you have to go one past the last number you want Then, you can cast it as a set() object: >>> b = set(a) Does that help? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Tue Aug 17 10:44:48 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 17 Aug 2010 07:44:48 -0700 Subject: [Numpy-discussion] set operation In-Reply-To: References: <29449267.post@talk.nabble.com> Message-ID: It should be noted that the 'set' type is a basic Python type, and not a NumPy specific type. On Tue, Aug 17, 2010 at 7:30 AM, Benjamin Root wrote: > On Tue, Aug 17, 2010 at 9:11 AM, gerardob wrote: > >> >> I would like to create a set composed of the numbers 2,3,4,...49,50,51. >> How >> do i do? >> Thanks. >> > > Gerard, > > Well, you can easily create a list of those numbers like so: > > >>> a = range(2, 52) # note, you have to go one past the last number you > want > > Then, you can cast it as a set() object: > > >>> b = set(a) > > Does that help? > Ben Root > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Tue Aug 17 11:08:49 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 17 Aug 2010 10:08:49 -0500 Subject: [Numpy-discussion] array manipulation In-Reply-To: References: Message-ID: <4C6AA601.4070801@enthought.com> Alex Ter-Sarkissov wrote: > hi, this is probably a very silly question, but I can't get my hear > around it unfortunately( > > I have an array (say, mat=rand(3,5)) from which I 'pull out' a row > (say, s1=mat[1,]). The problem is, the shape of this row s1 is not > [1,5], as I would expect, but rather [5,], which means that I can't, > for example, concateante mat and s1 rowwise. > You can use a trivial slice: In [9]: a Out[9]: array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14]]) In [10]: a[1] # This is a 1D array. Out[10]: array([5, 6, 7, 8, 9]) In [11]: a[1:2] # This is a 2D array. Out[11]: array([[5, 6, 7, 8, 9]]) Warren > > > thanks for the help > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Tue Aug 17 11:23:26 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 17 Aug 2010 23:23:26 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 Message-ID: I am pleased to announce the availability of the second beta of NumPy 1.5.0. This will be the first NumPy release to include support for Python 3, as well as for Python 2.7. Please try this beta and report any problems on the NumPy mailing list. Especially with Python 3 testing will be very useful. On Linux and OS X building from source should be straightforward, for Windows a binary installer is provided. There is one important known issue on Windows left, in fromfile and tofile (ticket 1583). Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ Enjoy, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From amcmorl at gmail.com Tue Aug 17 11:30:42 2010 From: amcmorl at gmail.com (Angus McMorland) Date: Tue, 17 Aug 2010 11:30:42 -0400 Subject: [Numpy-discussion] fancy indexing question Message-ID: Hi all, I'm having a bit of a brain block about the following fancy indexing question, and am hoping someone can point me in the right direction. I have a 3-d array `a` that I want to reorder along its 0th dimension differently for each 1st dimension, with the indices given in a 2-d array `ord`. I can achieve this as follows a = np.random.randint(10, size=(4,3,2)) ord = np.array([[3,2,1,0],[0,2,1,3],[3,0,2,1]]).T b = a[ord][:,np.eye(3, dtype=bool),:] but it seems like it should be possible to do this in one indexing step and avoid the intermediate and the eye call. Any suggestions gratefully received. Thanks, Angus. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From alan.isaac at gmail.com Tue Aug 17 12:04:45 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 17 Aug 2010 12:04:45 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: <4C6AB31D.2040707@gmail.com> Two warnings remain; no failures. Details below. Alan Isaac (Python 2.7 on Vista) Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test() Running unit tests for numpy NumPy version 1.5.0b2 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] nose version 0.11.0 .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ......................K................................................................... .......................................................................................... .....................................................................K.................... ......................................K..K.....Warning: invalid value encountered in exp ........Warning: invalid value encountered in absolute .Warning: invalid value encountered in absolute ................K...SK.S.......S.......................................................... ..................S....................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ...K.........K............................................................................ .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ...S...................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... .......................................................................................... ............................. ---------------------------------------------------------------------- Ran 2424 tests in 16.548s OK (KNOWNFAIL=8, SKIP=5) From pgmdevlist at gmail.com Tue Aug 17 12:07:00 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 17 Aug 2010 12:07:00 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6AB31D.2040707@gmail.com> References: <4C6AB31D.2040707@gmail.com> Message-ID: <4D9E4854-8ADD-4AFA-A4B1-308353ED7500@gmail.com> On Aug 17, 2010, at 12:04 PM, Alan G Isaac wrote: > Two warnings remain; no failures. > Details below. > Alan Isaac > (Python 2.7 on Vista) What does a np.ma.test() give you ? From josef.pktd at gmail.com Tue Aug 17 12:18:28 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Aug 2010 12:18:28 -0400 Subject: [Numpy-discussion] fancy indexing question In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 11:30 AM, Angus McMorland wrote: > Hi all, > > I'm having a bit of a brain block about the following fancy indexing > question, and am hoping someone can point me in the right direction. I > have a 3-d array `a` that I want to reorder along its 0th dimension > differently for each 1st dimension, with the indices given in a 2-d > array `ord`. I can achieve this as follows > > a = np.random.randint(10, size=(4,3,2)) > ord = np.array([[3,2,1,0],[0,2,1,3],[3,0,2,1]]).T > b = a[ord][:,np.eye(3, dtype=bool),:] > > but it seems like it should be possible to do this in one indexing > step and avoid the intermediate and the eye call. Any suggestions > gratefully received. >>> a = np.random.randint(10, size=(4,3,2)) >>> ord = np.array([[3,2,1,0],[0,2,1,3],[3,0,2,1]]).T >>> b = a[ord][:,np.eye(3, dtype=bool),:] >>> (a[ord, np.arange(3), :] == b).all() True TDD without looking to closely what it is Josef > > Thanks, > > Angus. > -- > AJC McMorland > Post-doctoral research fellow > Neurobiology, University of Pittsburgh > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Nikolaus at rath.org Tue Aug 17 11:53:02 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 17 Aug 2010 11:53:02 -0400 Subject: [Numpy-discussion] Find insertion point Message-ID: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> Hello, I want to find the first i such that x[i] < y and x[i+1] >= y. Is there a way to do this without using a Python loop? I can't use np.searchsorted(), because my x array crosses y several times. Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From alan.isaac at gmail.com Tue Aug 17 12:27:33 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 17 Aug 2010 12:27:33 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4D9E4854-8ADD-4AFA-A4B1-308353ED7500@gmail.com> References: <4C6AB31D.2040707@gmail.com> <4D9E4854-8ADD-4AFA-A4B1-308353ED7500@gmail.com> Message-ID: <4C6AB875.3080907@gmail.com> On 8/17/2010 12:07 PM, Pierre GM wrote: > What does a np.ma.test() give you ? No problems. (Aren't those always run?) Alan >>> np.ma.test() Running unit tests for numpy.ma NumPy version 1.5.0b2 NumPy is installed in C:\Python27\lib\site-packages\numpy Python version 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] nose version 0.11.0 .......................................................................................... .......................................................................................... .......................................................................................... ............................ ---------------------------------------------------------------------- Ran 298 tests in 1.588s OK From bsouthey at gmail.com Tue Aug 17 12:31:44 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 17 Aug 2010 11:31:44 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 10:23 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the second beta of NumPy 1.5.0. > This will be the first NumPy release to include support for Python 3, as > well as for Python 2.7. > > Please try this beta and report any problems on the NumPy mailing list. > Especially with Python 3 testing will be very useful. On Linux and OS X > building from source should be straightforward, for Windows a binary > installer is provided. There is one important known issue on Windows left, > in fromfile and tofile (ticket 1583). > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > > Enjoy, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Hi, How do you get nose installed under Windows with Python3.1 such as a single package or command? The binary installed with the 32-bit install of Python3.1 under Windows 7 . My 32-bit Python 2.6 version under Windows 7 passed fine. (I'll test my Linux versions when I get back to them.) Bruce From alan.isaac at gmail.com Tue Aug 17 12:35:13 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 17 Aug 2010 12:35:13 -0400 Subject: [Numpy-discussion] Find insertion point In-Reply-To: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> References: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> Message-ID: <4C6ABA41.2000605@gmail.com> On 8/17/2010 11:53 AM, Nikolaus Rath wrote: > I want to find the first i such that x[i]< y and x[i+1]>= y. Is there > a way to do this without using a Python loop? argmax? (to get i+1): >>> d = np.linspace(0,10,101) >>> x = np.sin(d) >>> np.argmax(x>=0.5) 6 fwiw, Alan Isaac From lane at brooks.nu Tue Aug 17 12:39:47 2010 From: lane at brooks.nu (Lane Brooks) Date: Tue, 17 Aug 2010 10:39:47 -0600 Subject: [Numpy-discussion] Find insertion point In-Reply-To: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> References: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> Message-ID: <4C6ABB53.7010507@brooks.nu> On 08/17/2010 09:53 AM, Nikolaus Rath wrote: > Hello, > > I want to find the first i such that x[i]< y and x[i+1]>= y. Is there > a way to do this without using a Python loop? > > I can't use np.searchsorted(), because my x array crosses y several > times. > > > Best, > > -Nikolaus > i = numpy.nonzero(x>=y)[0][0] From warren.weckesser at enthought.com Tue Aug 17 12:42:13 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 17 Aug 2010 11:42:13 -0500 Subject: [Numpy-discussion] Find insertion point In-Reply-To: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> References: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> Message-ID: <4C6ABBE5.6000104@enthought.com> Nikolaus Rath wrote: > Hello, > > I want to find the first i such that x[i] < y and x[i+1] >= y. Is there > a way to do this without using a Python loop? > > I can't use np.searchsorted(), because my x array crosses y several > times. > > > Best, > > -Nikolaus > > Here's one way: In [32]: x Out[32]: array([10, 3, 2, 5, 8, 7, 6, 5, 4, 7, 11]) In [33]: y = 4.1 In [34]: np.where((x[:-1] < y) & (x[1:] > y))[0][0] Out[34]: 2 In [35]: y = 9.5 In [36]: np.where((x[:-1] < y) & (x[1:] > y))[0][0] Out[36]: 9 (The second index [0] assumes that at least one such point is found.) Warren From pgmdevlist at gmail.com Tue Aug 17 12:43:41 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 17 Aug 2010 12:43:41 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6AB875.3080907@gmail.com> References: <4C6AB31D.2040707@gmail.com> <4D9E4854-8ADD-4AFA-A4B1-308353ED7500@gmail.com> <4C6AB875.3080907@gmail.com> Message-ID: On Aug 17, 2010, at 12:27 PM, Alan G Isaac wrote: > On 8/17/2010 12:07 PM, Pierre GM wrote: >> What does a np.ma.test() give you ? > > No problems. (Aren't those always run?) > Alan Yes, but I just wanted to check whether I was to blame for not having ported some changes to the 1.5.x branch or not. So I'm in the clear... From morph at debian.org Tue Aug 17 12:51:58 2010 From: morph at debian.org (Sandro Tosi) Date: Tue, 17 Aug 2010 18:51:58 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: Hello, On Tue, Aug 17, 2010 at 17:23, Ralf Gommers wrote: > I am pleased to announce the availability of the second beta of NumPy 1.5.0. > This will be the first NumPy release to include support for Python 3, as > well as for Python 2.7. > > Please try this beta and report any problems on the NumPy mailing list. > Especially with Python 3 testing will be very useful. On Linux and OS X > building from source should be straightforward, for Windows a binary > installer is provided. There is one important known issue on Windows left, > in fromfile and tofile (ticket 1583). (on a Debian system) I tested the build with python3.1, and it builds fine (it seems) but I couldn't run numpy.test() since we still don't have a nose for python3. anyhow the basic "print(numpy.__version__)" worked. On python2.6 and 2.5: Running unit tests for numpy NumPy version 1.5.0b2 NumPy is installed in /home/morph/deb/tmp/numpy-1.5.0b2/inst/lib/python2.6/site-packages/numpy Python version 2.6.5+ (release26-maint, Jul 6 2010, 12:58:20) [GCC 4.4.4] nose version 0.11.1 ... Ran 2966 tests in 16.129s OK (KNOWNFAIL=4, SKIP=4) Running unit tests for numpy NumPy version 1.5.0b2 NumPy is installed in /home/morph/deb/tmp/numpy-1.5.0b2/inst/lib/python2.5/site-packages/numpy Python version 2.5.5 (r255:77872, Apr 21 2010, 08:40:04) [GCC 4.4.3] nose version 0.11.1 ... Ran 2952 tests in 15.670s OK (KNOWNFAIL=4, SKIP=4) Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From mdroe at stsci.edu Tue Aug 17 13:27:45 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 17 Aug 2010 13:27:45 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: <4C6AC691.4000504@stsci.edu> I'm getting one unit test error on SunOS 5.8 UltraSPARC 5, with Sun WorkShop 6 update 2 C 5.3 Patch 111679-14 2004/02/20: ====================================================================== ERROR: test_ldexp (test_umath.TestLdexp) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mdroe/usr/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 388, in test_ldexp assert_almost_equal(ncu.ldexp(np.array(2., np.longdouble), np.array(3, np.int16)), 16.) TypeError: function not supported for these types, and can't coerce safely to supported types Any thoughts? Mike On 08/17/2010 11:23 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the second beta of NumPy > 1.5.0. This will be the first NumPy release to include support for > Python 3, as well as for Python 2.7. > > Please try this beta and report any problems on the NumPy mailing > list. Especially with Python 3 testing will be very useful. On Linux > and OS X building from source should be straightforward, for Windows a > binary installer is provided. There is one important known issue on > Windows left, in fromfile and tofile (ticket 1583). > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > > > Enjoy, > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Tue Aug 17 14:15:24 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 17 Aug 2010 14:15:24 -0400 Subject: [Numpy-discussion] Find insertion point In-Reply-To: <4C6ABA41.2000605@gmail.com> (Alan G. Isaac's message of "Tue, 17 Aug 2010 12:35:13 -0400") References: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> <4C6ABA41.2000605@gmail.com> Message-ID: <87sk2dksw3.fsf@inspiron.ap.columbia.edu> Alan G Isaac writes: > On 8/17/2010 11:53 AM, Nikolaus Rath wrote: >> I want to find the first i such that x[i]< y and x[i+1]>= y. Is there >> a way to do this without using a Python loop? > > > argmax? (to get i+1): > >>> d = np.linspace(0,10,101) > >>> x = np.sin(d) > >>> np.argmax(x>=0.5) > 6 Not working for e.g. x = [4,3,2,3,4,5] and y=3.5. Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From Nikolaus at rath.org Tue Aug 17 14:17:35 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 17 Aug 2010 14:17:35 -0400 Subject: [Numpy-discussion] Find insertion point In-Reply-To: <4C6ABB53.7010507@brooks.nu> (Lane Brooks's message of "Tue, 17 Aug 2010 10:39:47 -0600") References: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> <4C6ABB53.7010507@brooks.nu> Message-ID: <87pqxhkssg.fsf@inspiron.ap.columbia.edu> Lane Brooks writes: > On 08/17/2010 09:53 AM, Nikolaus Rath wrote: >> Hello, >> >> I want to find the first i such that x[i]< y and x[i+1]>= y. Is there >> a way to do this without using a Python loop? >> >> I can't use np.searchsorted(), because my x array crosses y several >> times. >> >> >> Best, >> >> -Nikolaus >> > > > i = numpy.nonzero(x>=y)[0][0] I'm afraid that also fails the x = [1,2,3,2,3,4], y=3 test. -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From Nikolaus at rath.org Tue Aug 17 14:21:16 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 17 Aug 2010 14:21:16 -0400 Subject: [Numpy-discussion] Find insertion point In-Reply-To: <4C6ABBE5.6000104@enthought.com> (Warren Weckesser's message of "Tue, 17 Aug 2010 11:42:13 -0500") References: <87vd79kzhd.fsf@inspiron.ap.columbia.edu> <4C6ABBE5.6000104@enthought.com> Message-ID: <87mxslksmb.fsf@inspiron.ap.columbia.edu> Warren Weckesser writes: > Nikolaus Rath wrote: >> Hello, >> >> I want to find the first i such that x[i] < y and x[i+1] >= y. Is there >> a way to do this without using a Python loop? >> >> I can't use np.searchsorted(), because my x array crosses y several >> times. >> > > In [34]: np.where((x[:-1] < y) & (x[1:] > y))[0][0] > Out[34]: 2 Neat, thanks! -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From bsouthey at gmail.com Tue Aug 17 14:23:43 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 17 Aug 2010 13:23:43 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: <4C6AD3AF.5090603@gmail.com> On 08/17/2010 11:51 AM, Sandro Tosi wrote: > Hello, > > On Tue, Aug 17, 2010 at 17:23, Ralf Gommers wrote: >> I am pleased to announce the availability of the second beta of NumPy 1.5.0. >> This will be the first NumPy release to include support for Python 3, as >> well as for Python 2.7. >> >> Please try this beta and report any problems on the NumPy mailing list. >> Especially with Python 3 testing will be very useful. On Linux and OS X >> building from source should be straightforward, for Windows a binary >> installer is provided. There is one important known issue on Windows left, >> in fromfile and tofile (ticket 1583). > (on a Debian system) I tested the build with python3.1, and it builds > fine (it seems) but I couldn't run numpy.test() since we still don't > have a nose for python3. anyhow the basic "print(numpy.__version__)" > worked. > > On python2.6 and 2.5: > > Running unit tests for numpy > NumPy version 1.5.0b2 > NumPy is installed in > /home/morph/deb/tmp/numpy-1.5.0b2/inst/lib/python2.6/site-packages/numpy > Python version 2.6.5+ (release26-maint, Jul 6 2010, 12:58:20) [GCC 4.4.4] > nose version 0.11.1 > ... > Ran 2966 tests in 16.129s > > OK (KNOWNFAIL=4, SKIP=4) > > Running unit tests for numpy > NumPy version 1.5.0b2 > NumPy is installed in > /home/morph/deb/tmp/numpy-1.5.0b2/inst/lib/python2.5/site-packages/numpy > Python version 2.5.5 (r255:77872, Apr 21 2010, 08:40:04) [GCC 4.4.3] > nose version 0.11.1 > ... > Ran 2952 tests in 15.670s > > OK (KNOWNFAIL=4, SKIP=4) > > Regards, On my 64-bit Fedora system all the following Python versions pass all the tests: Python2.4 Python2.5 Python2.6 Python2.7 Python3.1 Python3.2 alpha There appears to a common warning that, in at least using Python2.7, comes from this test: Tests masked_unary_operation ... Warning: divide by zero encountered in log ok It is really great to have both Python 3.1 and 3.2 working! Bruce From charlesr.harris at gmail.com Tue Aug 17 15:08:05 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 Aug 2010 13:08:05 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6AC691.4000504@stsci.edu> References: <4C6AC691.4000504@stsci.edu> Message-ID: On Tue, Aug 17, 2010 at 11:27 AM, Michael Droettboom wrote: > I'm getting one unit test error on SunOS 5.8 UltraSPARC 5, with Sun > WorkShop 6 update 2 C 5.3 Patch 111679-14 2004/02/20: > > ====================================================================== > ERROR: test_ldexp (test_umath.TestLdexp) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/mdroe/usr/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > line 388, in test_ldexp > assert_almost_equal(ncu.ldexp(np.array(2., np.longdouble), np.array(3, > np.int16)), 16.) > TypeError: function not supported for these types, and can't coerce safely > to supported types > > Any thoughts? > > Mike > > Does it worki for other SunOS versions? Can you pin down a bit more which types work and which not? I suspect that the longdouble may be a problem, but that's just because quad precision is mostly a Sun (Oracle?) thing. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Aug 17 15:38:02 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 17 Aug 2010 12:38:02 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: <4C6AE51A.40202@uci.edu> On 8/17/2010 8:23 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the second beta of NumPy > 1.5.0. This will be the first NumPy release to include support for > Python 3, as well as for Python 2.7. > > Please try this beta and report any problems on the NumPy mailing list. > Especially with Python 3 testing will be very useful. On Linux and OS X > building from source should be straightforward, for Windows a binary > installer is provided. There is one important known issue on Windows > left, in fromfile and tofile (ticket 1583). > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > > > Enjoy, > Ralf > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 (32 and 64 bit) still reports many (> 200) warnings and three known test failures/errors. Nothing serious, but it would be nice to clean up before the final release. The warnings are of the type "Warning: invalid value encountered in" for the functions reduce, fmax, fmin, logaddexp, maximum, greater, less_equal, greater_equal, absolute, and others. I do not see any of these warnings in the msvc9 builds of numpy 1.4.1. The following failure appears on 64 bit builds. I guess the test should be skipped but the platform_skip decorator introduced in changeset 8648 does not take effect because "(np.exp(complex(np.inf, 0)).imag != 0)" correctly evaluates to False. ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\numpy\testing\decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "X:\Python27-x64\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 242, in test_special_values assert_almost_equal(np.log(x), y) File "X:\Python27-x64\lib\site-packages\numpy\testing\utils.py", line 443, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: array([ nan+2.35619449j]) DESIRED: (inf+2.356194490192345j) >> raise AssertionError('\nArrays are not almost equal\n ACTUAL: array([ nan+2.35619449j])\n DESIRED: (inf+2.356194490192345j)') Similar, this failure appears on 32 bit and could be skipped or marked as known fail: ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\numpy\testing\decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "X:\Python27\lib\site-packages\numpy\core\tests\test_umath_complex.py", line 162, in test_special_values self.assertRaises(FloatingPointError, np.log, x) AssertionError: FloatingPointError not raised This error can be marked as known for 64 bit builds for Python 2.x (a patch is attached): ====================================================================== ERROR: Ticket #99 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\numpy\testing\decorators.py", line 215, in knownfailer return f(*args, **kwargs) File "X:\Python27-x64\lib\site-packages\numpy\core\tests\test_regression.py", line 146, in test_intp np.intp('0x' + 'f'*i_width,16) TypeError: function takes at most 1 argument (2 given) The Python 3.1 builds show no additional failures/errors, specifically not the ones mentioned in ticket 1583. -- Christoph -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test_regression.diff URL: From charlesr.harris at gmail.com Tue Aug 17 16:02:01 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 Aug 2010 14:02:01 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6AE51A.40202@uci.edu> References: <4C6AE51A.40202@uci.edu> Message-ID: On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke wrote: > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > >> I am pleased to announce the availability of the second beta of NumPy >> 1.5.0. This will be the first NumPy release to include support for >> Python 3, as well as for Python 2.7. >> >> Please try this beta and report any problems on the NumPy mailing list. >> Especially with Python 3 testing will be very useful. On Linux and OS X >> building from source should be straightforward, for Windows a binary >> installer is provided. There is one important known issue on Windows >> left, in fromfile and tofile (ticket 1583). >> >> Binaries, sources and release notes can be found at >> https://sourceforge.net/projects/numpy/files/ >> >> >> Enjoy, >> Ralf >> >> > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 (32 and 64 > bit) still reports many (> 200) warnings and three known test > failures/errors. Nothing serious, but it would be nice to clean up before > the final release. > > The warnings are of the type "Warning: invalid value encountered in" for > the functions reduce, fmax, fmin, logaddexp, maximum, greater, less_equal, > greater_equal, absolute, and others. I do not see any of these warnings in > the msvc9 builds of numpy 1.4.1. > > The warnings were accidentally turned off for earlier versions of Numpy. I expect these warnings are related to nans and probably due to problems with isnan or some such. Can you take a closer look? The fmax function should be easy to check out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Aug 17 16:43:41 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 17 Aug 2010 15:43:41 -0500 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: References: Message-ID: <4C6AF47D.6070900@gmail.com> On 08/16/2010 10:00 PM, Charles R Harris wrote: > Hi All, > > I just added support for Legendre polynomials to numpy and I think the > numpy.polynomial name space is getting a bit crowded. Since most of > the current functions in that namespace are just used to implement the > Polynomial, Chebyshev, and Legendre classes I'm thinking of only > importing those classes by default and leaving the other functions to > explicit imports. Of course I will have to fix the examples and maybe > some other users will be inconvenienced by the change. But with 2.0.0 > in the works this might be a good time to do this. Thoughts? > > Chuck While I don't know a lot about this so things will be easily off base. In looking at the names, I did see many names that seem identical except that these work just with one type of polynomial. Obviously cheb2poly and poly2cheb are the conversion between the polynomial and Chebyshev types - similarly leg2poly and poly2leg for the polynomial and Legendre classes. But none between Chebyshev and Legendre classes. Would it make more sense to create a single conversion function to change one type into another instead of the current 6 possibilities? Similarily there are obviously a very similar functions that just work with one polynomial type so the functionality is duplicated across each class that could be a single function each: chebadd legadd polyadd chebder legder polyder chebdiv legdiv polydiv chebdomain legdomain polydomain chebfit legfit polyfit chebfromroots legfromroots polyfromroots chebint legint polyint chebline legline polyline chebmul legmul polymul chebmulx legmulx polymulx chebone legone polyone chebroots legroots polyroots chebsub legsub polysub chebtrim legtrim polytrim chebval legval polyval chebvander legvander polyvander chebx legx polyx chebzero legzero polyzero However, I doubt that is worth the work if the overall amount of code is not reduced. For example, if you create a overall function that just calls the appropriate add function for that type of polynomial then I do not see any advantage in doing so just to reduce the namespace. If you can argue that is very beneficial to the user of polynomial functions then that could put a different spin on doing that. While I would have to check more carefully (as I don't have time now), aren't chebadd, legadd and polyadd essentially the same function? That is, can you send a Legendre polynomial to the same Chebysnev function and get the same answer back? If so then these functions should be collapsed into one for numpy 2.0. Just my 1 cent on that, Bruce From charlesr.harris at gmail.com Tue Aug 17 17:34:31 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 Aug 2010 15:34:31 -0600 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: <4C6AF47D.6070900@gmail.com> References: <4C6AF47D.6070900@gmail.com> Message-ID: On Tue, Aug 17, 2010 at 2:43 PM, Bruce Southey wrote: > On 08/16/2010 10:00 PM, Charles R Harris wrote: > > Hi All, > > > > I just added support for Legendre polynomials to numpy and I think the > > numpy.polynomial name space is getting a bit crowded. Since most of > > the current functions in that namespace are just used to implement the > > Polynomial, Chebyshev, and Legendre classes I'm thinking of only > > importing those classes by default and leaving the other functions to > > explicit imports. Of course I will have to fix the examples and maybe > > some other users will be inconvenienced by the change. But with 2.0.0 > > in the works this might be a good time to do this. Thoughts? > > > > Chuck > While I don't know a lot about this so things will be easily off base. > > In looking at the names, I did see many names that seem identical except > that these work just with one type of polynomial. > > Obviously cheb2poly and poly2cheb are the conversion between the > polynomial and Chebyshev types - similarly leg2poly and poly2leg for the > polynomial and Legendre classes. But none between Chebyshev and Legendre > classes. Would it make more sense to create a single conversion function > to change one type into another instead of the current 6 possibilities? > > The class types can be converted to each other, with an optional change of domain, using the convert method, i.e., if p is an instance of Legendre p.convert(kind=Chebyshev) will do the conversion to a Chebyshev series.. The classes don't actually use the *2* functions, oddly enough ;) > Similarily there are obviously a very similar functions that just work > with one polynomial type so the functionality is duplicated across each > class that could be a single function each: > chebadd legadd polyadd > chebder legder polyder > chebdiv legdiv polydiv > chebdomain legdomain polydomain > chebfit legfit polyfit > chebfromroots legfromroots polyfromroots > chebint legint polyint > chebline legline polyline > chebmul legmul polymul > chebmulx legmulx polymulx > chebone legone polyone > chebroots legroots polyroots > chebsub legsub polysub > chebtrim legtrim polytrim > chebval legval polyval > chebvander legvander polyvander > chebx legx polyx > chebzero legzero polyzero > > However, I doubt that is worth the work if the overall amount of code is > not reduced. For example, if you create a overall function that just > calls the appropriate add function for that type of polynomial then I do > not see any advantage in doing so just to reduce the namespace. > If you can argue that is very beneficial to the user of polynomial > functions then that could put a different spin on doing that. > > While I would have to check more carefully (as I don't have time now), > aren't chebadd, legadd and polyadd essentially the same function? > That is, can you send a Legendre polynomial to the same Chebysnev > function and get the same answer back? > If so then these functions should be collapsed into one for numpy 2.0. > > Yeah, the add and subtract functions are all the same along with the *trim functions. These things are all accessable through the classes ustng +/- and the trim and truncate methods. Which is why for normal work I think the classes are the way to go, the functions are just for implementing the classes and available in case someone wants to roll their own. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Tue Aug 17 17:42:54 2010 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 17 Aug 2010 17:42:54 -0400 Subject: [Numpy-discussion] numpy.concatenate slower than slice copying In-Reply-To: <20100817130310.GA12292@in.waw.pl> References: <20100817130310.GA12292@in.waw.pl> Message-ID: Yes, concatenate is doing other work under the covers. In short, in supports concatenating a list of arbitrary python sequences into an array and does checking on each element of the tuple to ensure it is valid to concatenate. On Tue, Aug 17, 2010 at 9:03 AM, Zbyszek Szmek wrote: > Hi, > this is a problem which came up when trying to replace a hand-written > array concatenation with a call to numpy.vstack: > for some array sizes, > > numpy.vstack(data) > > runs > 20% longer than a loop like > > alldata = numpy.empty((tlen, dim)) > for x in data: > step = x.shape[0] > alldata[pos:pos+step] = x > pos += step > > (example script attached) > > $ python del_cum3.py numpy 10000 10000 1 10 > problem size: (10000x10000) x 1 = 10^8 > 0.816s <------------------------------- numpy.concatentate of 10 arrays > 10000x10000 > > $ python del_cum3.py concat 10000 10000 1 10 > problem size: (10000x10000) x 1 = 10^8 > 0.642s <------------------------------- slice manipulation giving the same > result > > When the array size is reduced to 100x100 or so, the computation time goes > to 0, > so it seems that the dtype and dimension checking is negligible. > Does numpy.concatenate do some extra work? > > Thanks for any pointers, > Zbyszek > > PS. Architecture is amd64. > python2.6, numpy 1.3.0 > or > python3.1, numpy 2.0.0.dev / trunk at 8510 > give the same result. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashford at whisperpc.com Tue Aug 17 18:39:48 2010 From: ashford at whisperpc.com (ashford at whisperpc.com) Date: Tue, 17 Aug 2010 15:39:48 -0700 (PDT) Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <201008171343.44255.eg@fft.be> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <4C6A1992.6050608@silveregg.co.jp> <33559.66.146.167.66.1282022069.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> Message-ID: <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> Eloi, > please below a script that will build numpy using a relevant site.cfg for your > configuration (you need to update GOTODIR and LAPACKDIR and PYTHONDIR): > #copy site.cfg > cp ./site.cfg PYTHONDIR/lib/python2.6/site-packages/numpy/distutils/. I believe this needs a $ prior to PYTHONDIR. I tried this, and the benchmark still came in at 8.5S. Any ideas? How long should the following benchmark take on a Core-i7 930, with Atlas or Goto? time python -c "import numpy as N; a=N.random.randn(1000, 1000); N.dot(a, a)" Thank you. Peter Ashford From scott.p.macdonald at gmail.com Tue Aug 17 20:00:06 2010 From: scott.p.macdonald at gmail.com (Scott MacDonald) Date: Tue, 17 Aug 2010 17:00:06 -0700 Subject: [Numpy-discussion] Problem with loadtxt Message-ID: Hi, I have a text file called 'blah' that contains only the following line: 5399354557888517312,5399354557888517312 I want to load these into a numpy array as unit64's. The following script demonstrates my problem: import numpy as np with open('blah', 'r') as f: x = np.loadtxt(f, delimiter=',', dtype=(np.uint64, np.uint64)) with open('blah', 'r') as f: for line in f: print line print x The output I get is: 5399354557888517312,5399354557888517312 [5399354557888517120 5399354557888517120] Am I doing something wrong? Thanks for your help, Scott -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Aug 17 20:07:50 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 18 Aug 2010 08:07:50 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 12:31 AM, Bruce Southey wrote: > On Tue, Aug 17, 2010 at 10:23 AM, Ralf Gommers > wrote: > > I am pleased to announce the availability of the second beta of NumPy > 1.5.0. > > This will be the first NumPy release to include support for Python 3, as > > well as for Python 2.7. > > > > Please try this beta and report any problems on the NumPy mailing list. > > Especially with Python 3 testing will be very useful. On Linux and OS X > > building from source should be straightforward, for Windows a binary > > installer is provided. There is one important known issue on Windows > left, > > in fromfile and tofile (ticket 1583). > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > > > Enjoy, > > Ralf > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > Hi, > How do you get nose installed under Windows with Python3.1 such as a > single package or command? There is a nose branch that works with py3k here: http://bitbucket.org/jpellerin/nose3/. Just install with "python setup.py install". I will try not to forget to mention this in the next announcement. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Aug 17 20:10:51 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 18 Aug 2010 08:10:51 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AC691.4000504@stsci.edu> Message-ID: On Wed, Aug 18, 2010 at 3:08 AM, Charles R Harris wrote: > > > On Tue, Aug 17, 2010 at 11:27 AM, Michael Droettboom wrote: > >> I'm getting one unit test error on SunOS 5.8 UltraSPARC 5, with Sun >> WorkShop 6 update 2 C 5.3 Patch 111679-14 2004/02/20: >> >> ====================================================================== >> ERROR: test_ldexp (test_umath.TestLdexp) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/home/mdroe/usr/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", >> line 388, in test_ldexp >> assert_almost_equal(ncu.ldexp(np.array(2., np.longdouble), np.array(3, >> np.int16)), 16.) >> TypeError: function not supported for these types, and can't coerce safely >> to supported types >> >> Any thoughts? >> >> Mike >> >> > Does it worki for other SunOS versions? Can you pin down a bit more which > types work and which not? I suspect that the longdouble may be a problem, > but that's just because quad precision is mostly a Sun (Oracle?) thing. > It's http://projects.scipy.org/numpy/ticket/1499. Fixed for Windows in r8625, can you check if the same fix works on SunOS? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Tue Aug 17 20:13:55 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 17 Aug 2010 19:13:55 -0500 Subject: [Numpy-discussion] Problem with loadtxt In-Reply-To: References: Message-ID: <4C6B25C3.40204@enthought.com> Scott MacDonald wrote: > Hi, > > I have a text file called 'blah' that contains only the following line: > > 5399354557888517312,5399354557888517312 > > I want to load these into a numpy array as unit64's. The following > script demonstrates my problem: > > import numpy as np > > with open('blah', 'r') as f: > x = np.loadtxt(f, delimiter=',', dtype=(np.uint64, np.uint64)) > > with open('blah', 'r') as f: > for line in f: > print line > > print x > > > The output I get is: > > 5399354557888517312,5399354557888517312 > > [5399354557888517120 5399354557888517120] > > > Am I doing something wrong? > I don't know the full story, but it looks like the conversion from text to an integer might go through a float: In [27]: int(float(5399354557888517312)) Out[27]: 5399354557888517120L You can change this by explicitly giving a converter to loadtxt: In [31]: with open('blah', 'r') as f: ....: x = np.loadtxt(f, delimiter=',', dtype=(np.uint64, np.uint64), converters={0:int}) ....: ....: In [32]: x Out[32]: array([5399354557888517312, 5399354557888517120], dtype=uint64) Note that the first column, for which I explicitly told it to use int() as the converter, is correct. Warren > Thanks for your help, > Scott > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From david at silveregg.co.jp Tue Aug 17 21:00:40 2010 From: david at silveregg.co.jp (David) Date: Wed, 18 Aug 2010 10:00:40 +0900 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <201008171343.44255.eg@fft.be> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <4C6A1992.6050608@silveregg.co.jp> <33559.66.146.167.66.1282022069.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> Message-ID: <4C6B30B8.3020300@silveregg.co.jp> On 08/17/2010 08:43 PM, Eloi Gaudry wrote: > Peter, > > please below a script that will build numpy using a relevant site.cfg for your configuration (you need to update GOTODIR and LAPACKDIR and PYTHONDIR): > > #!/bin/sh > > #BLAS/LAPACK configuration file > echo "[blas]"> ./site.cfg > echo "library_dirs = GOTODIR">> ./site.cfg > echo "blas_libs = goto2_nehalemp-r1.13">> ./site.cfg > echo "[lapack]">> ./site.cfg > echo "library_dirs = LAPACKDIR">> ./site.cfg > echo "lapack_libs = lapack">> ./site.cfg > > #compilation variables > export CC=gcc > export F77=gfortran > export F90=gfortran > export F95=gfortran > export LDFLAGS="-shared -Wl,-rpath=\'\$ORIGIN/../../../..\'" > export CFLAGS="-O1 -pthread" > export FFLAGS="-O2" > > #build > python setup.py config > python setup.py build > python setup.py install > > #copy site.cfg > cp ./site.cfg PYTHONDIR/lib/python2.6/site-packages/numpy/distutils/. This should be PYTHONPATH, not PYTHONDIR. Also, on 64 bits, you need -fPIC in every *FLAGS variables. cheers, David From david at silveregg.co.jp Tue Aug 17 21:02:21 2010 From: david at silveregg.co.jp (David) Date: Wed, 18 Aug 2010 10:02:21 +0900 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <4C6A1992.6050608@silveregg.co.jp> <33559.66.146.167.66.1282022069.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> Message-ID: <4C6B311D.9020909@silveregg.co.jp> On 08/18/2010 07:39 AM, ashford at whisperpc.com wrote: > Eloi, > >> please below a script that will build numpy using a relevant site.cfg for your >> configuration (you need to update GOTODIR and LAPACKDIR and PYTHONDIR): > >> #copy site.cfg >> cp ./site.cfg PYTHONDIR/lib/python2.6/site-packages/numpy/distutils/. > > I believe this needs a $ prior to PYTHONDIR. > > I tried this, and the benchmark still came in at 8.5S. > > Any ideas? How long should the following benchmark take on a Core-i7 930, > with Atlas or Goto? > > time python -c "import numpy as N; a=N.random.randn(1000, 1000); N.dot(a, a)" Do you have a _dotblas.so file ? We only support _dotblas linked against atlas AFAIK, which means goto won't be used in that case. To check which libraries are used by your extensions, you should use ldd on the .so files (for example ldd .../numpy/linalg/lapack_lite.so). cheers, David From faltet at pytables.org Tue Aug 17 21:06:51 2010 From: faltet at pytables.org (Francesc Alted) Date: Wed, 18 Aug 2010 02:06:51 +0100 Subject: [Numpy-discussion] numpy.concatenate slower than slice copying In-Reply-To: <20100817130310.GA12292@in.waw.pl> References: <20100817130310.GA12292@in.waw.pl> Message-ID: Hey Zbyszek, 2010/8/17, Zbyszek Szmek : > Hi, > this is a problem which came up when trying to replace a hand-written > array concatenation with a call to numpy.vstack: > for some array sizes, > > numpy.vstack(data) > > runs > 20% longer than a loop like > > alldata = numpy.empty((tlen, dim)) > for x in data: > step = x.shape[0] > alldata[pos:pos+step] = x > pos += step > > (example script attached) [clip] I was curious on what is happening here, so after some profiling with cachegrind, I've come to the conclusion that `numpy.concatenate` is using the `memcpy` system call so as to copy data from sources to recipient. On his hand, your `concat` function is making use of the `__setitem__` method of ndarray, which does not use `memcpy` (this is probably due to the fact that it has to deal with strides). Now, it turns out that `memcpy` may be not optimal for every platform, and a direct fetch and assign approach could be sometimes faster. My guess is that this is what is happening in your case. On my machine, running latest Ubuntu Linux, I'm not seeing this difference though: faltet at ubuntu:~/carray$ python bench/concat.py numpy 1000 1000 10 3 problem size: (1000x1000) x 10 = 10^7 0.247s faltet at ubuntu:~/carray$ python bench/concat.py concat 1000 1000 10 3 problem size: (1000x1000) x 10 = 10^7 0.246s and neither when running Windows (XP): C:\tmp>python del_cum3.py numpy 10000 1000 1 10 problem size: (10000x1000) x 1 = 10^7 0.227s C:\tmp>python del_cum3.py concat 10000 1000 1 10 problem size: (10000x1000) x 1 = 10^7 0.223s Coincidentally, I've been lately working out a proof of concept for an array that can hold data in-memory in compressed state (using the high-performance Blosc compressor under the hood). This object (I'm calling it ``carray`` for the time being) can also be `append`-ed with additional data, so it can be used in this concatenation use case. So, I've setup a new benchmark based in your script (I called it concat.py) and tried it out with your problem. Here are the results for my netbook wearing a humble Intel Atom processor. First, the figures for the initial `numpy.concatenate` and `concat` styles: faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py numpy 1000000 10 3 problem size: (1000000) x 10 = 10^7 time for concat: 0.228s size of the final container: 76.294 MB faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py concat 1000000 10 3 problem size: (1000000) x 10 = 10^7 time for concat: 0.230s size of the final container: 76.294 MB Now the new method (carray) with compression level 1 (note the new parameter at the end of the command line): faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py carray 1000000 10 3 1 problem size: (1000000) x 10 = 10^7 time for concat: 0.186s size of the final container: 5.076 MB which is more than a 20% faster than `numpy.concatenate` or your `concat` method, while the space taken in memory is significantly lower (5.1 MB vs 76.3 MB; of course, I've chosen a very compressible dataset for this example ;-) Even if you tell Blosc not to use compression (Blosc level 0), I can still see a win here: faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py carray 1000000 10 3 0 problem size: (1000000) x 10 = 10^7 time for concat: 0.200s size of the final container: 77.001 MB which is 15% faster than the initial cases. However, note how space grows from the original 76.3 MB to 77.0 MB. This is because carray has to keep an internal buffer for accelerating the appending of small arrays; this is the main responsible of the space overhead. Finally, it is interesting to see the effect of forcing the use of a single thread instead of two (Atom has support for hyper-threading): faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py carray 1000000 10 3 0 problem size: (1000000) x 10 = 10^7 time for concat: 0.210s size of the final container: 77.001 MB which is still a 10% faster than plain `numpy.concatenate` (remember, based on `memcpy`). Why carray/Blosc is faster for this case is rather a mystery to me, but the effect is there. I have not yet released carray publicly, but in case you want to play with it, I've uploaded my current git repository to: http://www.pytables.org/download/preliminary/carray-0.1.dev.tar.gz Of course, carray is still pre-alpha, and it does not support multidimensional arrays and you cannot modify its contents (other than append new data) but still, it can be a lot of fun. Cheers! -- Francesc Alted From cournape at gmail.com Tue Aug 17 21:28:55 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 18 Aug 2010 10:28:55 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 12:23 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the second beta of NumPy 1.5.0. > This will be the first NumPy release to include support for Python 3, as > well as for Python 2.7. > > Please try this beta and report any problems on the NumPy mailing list. > Especially with Python 3 testing will be very useful. On Linux and OS X > building from source should be straightforward, for Windows a binary > installer is provided. There is one important known issue on Windows left, > in fromfile and tofile (ticket 1583). > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ I saw that the beta was put on pypi as well. I think we should not do that: easy_install will always the most recent version, and this means trouble. cheers, David From bsouthey at gmail.com Tue Aug 17 21:32:10 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 17 Aug 2010 20:32:10 -0500 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 7:07 PM, Ralf Gommers wrote: > > > On Wed, Aug 18, 2010 at 12:31 AM, Bruce Southey wrote: >> >> On Tue, Aug 17, 2010 at 10:23 AM, Ralf Gommers >> wrote: >> > I am pleased to announce the availability of the second beta of NumPy >> > 1.5.0. >> > This will be the first NumPy release to include support for Python 3, as >> > well as for Python 2.7. >> > >> > Please try this beta and report any problems on the NumPy mailing list. >> > Especially with Python 3 testing will be very useful. On Linux and OS X >> > building from source should be straightforward, for Windows a binary >> > installer is provided. There is one important known issue on Windows >> > left, >> > in fromfile and tofile (ticket 1583). >> > >> > Binaries, sources and release notes can be found at >> > https://sourceforge.net/projects/numpy/files/ >> > >> > Enjoy, >> > Ralf >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> >> Hi, >> How do you get nose installed under Windows with Python3.1 such as a >> single package or command? > > > There is a nose branch that works with py3k here: > http://bitbucket.org/jpellerin/nose3/. Just install with "python setup.py > install". > > I will try not to forget to mention this in the next announcement. > > Cheers, > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Okay, While I did not want to use hg, it was not very painful... It did result in 38 errors (given below) but I did not see anything that I would suggest a numpy error. All these seem to be Windows file access related problems which I did not see on the same machine: NumPy version 1.5.0b2 NumPy is installed in E:\Python26\lib\site-packages\numpy Python version 2.6.3 (r263rc1:75186, Oct 2 2009, 20:40:30) [MSC v.1500 32 bit (Intel)] nose version 0.11.1 So it could be nose related. Bruce Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.5.0b2 NumPy is installed in E:\Python31\lib\site-packages\numpy Python version 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit (Intel)] nose version 3.0.0 .........................................................................................................................................EEEEEEEEEEEEEEEEEEEEEEEEEE.EEEE..EEEEEEEE........................................................................................................................................................................................................................................................................................................................K......................................................K...........................................................................................................................................................................K..........................................................K..K..............................K...SK.S.......S............................................................................S..................................................................................................................................................................................................................................................................................................................................................................................................................................................K.........K.......................................................................................................................................................................................................................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ====================================================================== ERROR: test_multiarray.TestIO.test_ascii ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest self.test(*self.arg) File "E:\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1017, in test_ascii self._check_from('1 , 2 , 3 , 4', [1.,2.,3.,4.], sep=',') File "E:\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 980, in _check_from y = np.fromfile(self.filename, **kw) IOError: first argument must be an open file ====================================================================== ERROR: test_multiarray.TestIO.test_ascii ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 370, in tearDown try_run(self.inst, ('teardown', 'tearDown')) File "E:\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\util.py", line 469, in try_run return func() File "E:\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 918, in tearDown os.unlink(self.filename) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'e:\\users\\me\\appdata\\local\\temp\\tmp2ig8l6' ====================================================================== ERROR: test_multiarray.TestIO.test_binary ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest self.test(*self.arg) File "E:\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 999, in test_binary dtype='", line 1, in numpy.test() RuntimeError: lost sys.stdout >>> From scott.p.macdonald at gmail.com Tue Aug 17 22:15:41 2010 From: scott.p.macdonald at gmail.com (Scott MacDonald) Date: Tue, 17 Aug 2010 19:15:41 -0700 Subject: [Numpy-discussion] Problem with loadtxt In-Reply-To: <4C6B25C3.40204@enthought.com> References: <4C6B25C3.40204@enthought.com> Message-ID: Thanks Warren, that does solve the problem. I am still confused however by this: In [42]: c = StringIO("5399354557888517120") In [43]: np.loadtxt(c, dtype=np.uint64) Out[43]: array(5399354557888517120L, dtype=uint64) Does the above seem contradictory to you? Thanks, Scott On Tue, Aug 17, 2010 at 5:13 PM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > Scott MacDonald wrote: > > Hi, > > > > I have a text file called 'blah' that contains only the following line: > > > > 5399354557888517312,5399354557888517312 > > > > I want to load these into a numpy array as unit64's. The following > > script demonstrates my problem: > > > > import numpy as np > > > > with open('blah', 'r') as f: > > x = np.loadtxt(f, delimiter=',', dtype=(np.uint64, np.uint64)) > > > > with open('blah', 'r') as f: > > for line in f: > > print line > > > > print x > > > > > > The output I get is: > > > > 5399354557888517312,5399354557888517312 > > > > [5399354557888517120 5399354557888517120] > > > > > > Am I doing something wrong? > > > > I don't know the full story, but it looks like the conversion from text > to an integer might go through a float: > > In [27]: int(float(5399354557888517312)) > Out[27]: 5399354557888517120L > > > You can change this by explicitly giving a converter to loadtxt: > > In [31]: with open('blah', 'r') as f: > ....: x = np.loadtxt(f, delimiter=',', dtype=(np.uint64, > np.uint64), converters={0:int}) > ....: > ....: > > In [32]: x > Out[32]: array([5399354557888517312, 5399354557888517120], dtype=uint64) > > Note that the first column, for which I explicitly told it to use int() > as the converter, is correct. > > > Warren > > > > Thanks for your help, > > Scott > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Aug 17 23:11:44 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 17 Aug 2010 20:11:44 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AE51A.40202@uci.edu> Message-ID: <4C6B4F70.90301@uci.edu> On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke > wrote: > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > I am pleased to announce the availability of the second beta of > NumPy > 1.5.0. This will be the first NumPy release to include support for > Python 3, as well as for Python 2.7. > > Please try this beta and report any problems on the NumPy > mailing list. > Especially with Python 3 testing will be very useful. On Linux > and OS X > building from source should be straightforward, for Windows a binary > installer is provided. There is one important known issue on Windows > left, in fromfile and tofile (ticket 1583). > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > > > Enjoy, > Ralf > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 (32 > and 64 bit) still reports many (> 200) warnings and three known test > failures/errors. Nothing serious, but it would be nice to clean up > before the final release. > > The warnings are of the type "Warning: invalid value encountered in" > for the functions reduce, fmax, fmin, logaddexp, maximum, greater, > less_equal, greater_equal, absolute, and others. I do not see any of > these warnings in the msvc9 builds of numpy 1.4.1. > > > The warnings were accidentally turned off for earlier versions of Numpy. > I expect these warnings are related to nans and probably due to problems > with isnan or some such. Can you take a closer look? The fmax function > should be easy to check out. > > > > Chuck > Thanks for the hint. Warnings are issued in the test_umath test_*nan* functions. The problem can be condensed to this statement: >>> numpy.array([numpy.nan]) > 0 Warning: invalid value encountered in greater array([False], dtype=bool) When using msvc, ordered comparisons involving NaN raise an exception [1], i.e. set the 'invalid' x87 status bit, which leads to the warning being printed. I don't know if this violates IEEE 754 or C99 standards but it does not happen with the gcc builds. Maybe seterr(invalid='ignore') could be added to the test_*nan* functions? [1] http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx -- Christoph From charlesr.harris at gmail.com Tue Aug 17 23:52:59 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 Aug 2010 21:52:59 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6B4F70.90301@uci.edu> References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> Message-ID: On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke wrote: > > > On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke > > wrote: > > > > > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > > > I am pleased to announce the availability of the second beta of > > NumPy > > 1.5.0. This will be the first NumPy release to include support > for > > Python 3, as well as for Python 2.7. > > > > Please try this beta and report any problems on the NumPy > > mailing list. > > Especially with Python 3 testing will be very useful. On Linux > > and OS X > > building from source should be straightforward, for Windows a > binary > > installer is provided. There is one important known issue on > Windows > > left, in fromfile and tofile (ticket 1583). > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > > > > > Enjoy, > > Ralf > > > > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 (32 > > and 64 bit) still reports many (> 200) warnings and three known test > > failures/errors. Nothing serious, but it would be nice to clean up > > before the final release. > > > > The warnings are of the type "Warning: invalid value encountered in" > > for the functions reduce, fmax, fmin, logaddexp, maximum, greater, > > less_equal, greater_equal, absolute, and others. I do not see any of > > these warnings in the msvc9 builds of numpy 1.4.1. > > > > > > The warnings were accidentally turned off for earlier versions of Numpy. > > I expect these warnings are related to nans and probably due to problems > > with isnan or some such. Can you take a closer look? The fmax function > > should be easy to check out. > > > > > > > > Chuck > > > > > Thanks for the hint. Warnings are issued in the test_umath test_*nan* > functions. The problem can be condensed to this statement: > > >>> numpy.array([numpy.nan]) > 0 > Warning: invalid value encountered in greater > array([False], dtype=bool) > > > When using msvc, ordered comparisons involving NaN raise an exception > [1], i.e. set the 'invalid' x87 status bit, which leads to the warning > being printed. I don't know if this violates IEEE 754 or C99 standards > but it does not happen with the gcc builds. Maybe > seterr(invalid='ignore') could be added to the test_*nan* functions? > > [1] http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx > > The c standard is a bit flexible with respect to nans. For instance, if quiet nans aren't available macros that require quiet nans can't be implemented. But it seems to me that msvc is a bit short on the numerical side, as it has always been. Not only does the precise option disable bogus optimizations like x - x == 0 (good), it raises exceptions for ordered nan comparisons (bad). There doesn't seem to be any reasonable combination of features. I wonder if this "feature" is implemented by having signaling nans everywhere? Anyway, I don't know the best way to handle this. We can probably redo most of the code to avoid the comparisons in the nan case and that might be the way to go for 2.0, but for 1.5 a known fail might be appropriate. Thanks for looking into this. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 18 00:56:24 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 Aug 2010 22:56:24 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6B4F70.90301@uci.edu> References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> Message-ID: On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke wrote: > > > On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke > > wrote: > > > > > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > > > I am pleased to announce the availability of the second beta of > > NumPy > > 1.5.0. This will be the first NumPy release to include support > for > > Python 3, as well as for Python 2.7. > > > > Please try this beta and report any problems on the NumPy > > mailing list. > > Especially with Python 3 testing will be very useful. On Linux > > and OS X > > building from source should be straightforward, for Windows a > binary > > installer is provided. There is one important known issue on > Windows > > left, in fromfile and tofile (ticket 1583). > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > > > > > Enjoy, > > Ralf > > > > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 (32 > > and 64 bit) still reports many (> 200) warnings and three known test > > failures/errors. Nothing serious, but it would be nice to clean up > > before the final release. > > > > The warnings are of the type "Warning: invalid value encountered in" > > for the functions reduce, fmax, fmin, logaddexp, maximum, greater, > > less_equal, greater_equal, absolute, and others. I do not see any of > > these warnings in the msvc9 builds of numpy 1.4.1. > > > > > > The warnings were accidentally turned off for earlier versions of Numpy. > > I expect these warnings are related to nans and probably due to problems > > with isnan or some such. Can you take a closer look? The fmax function > > should be easy to check out. > > > > > > > > Chuck > > > > > Thanks for the hint. Warnings are issued in the test_umath test_*nan* > functions. The problem can be condensed to this statement: > > >>> numpy.array([numpy.nan]) > 0 > Warning: invalid value encountered in greater > array([False], dtype=bool) > > > When using msvc, ordered comparisons involving NaN raise an exception > [1], i.e. set the 'invalid' x87 status bit, which leads to the warning > being printed. I don't know if this violates IEEE 754 or C99 standards > but it does not happen with the gcc builds. Maybe > seterr(invalid='ignore') could be added to the test_*nan* functions? > > [1] http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx > > OK, this does seem to be the standard. For instance The isless macro determines whether its first argument is less than its second argument. The value of isless(x, y) is always equal to (x) < (y); however, unlike (x) < (y), isless(x, y) does not raise the ??invalid?? floating-point exception when x and y are unordered. There are other macros for the rest of the comparisons. Can you check if MSVC has these macros available? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Wed Aug 18 01:00:11 2010 From: david at silveregg.co.jp (David) Date: Wed, 18 Aug 2010 14:00:11 +0900 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> Message-ID: <4C6B68DB.9080604@silveregg.co.jp> On 08/18/2010 01:56 PM, Charles R Harris wrote: > > > On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke > wrote: > > > > On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke > > > >> wrote: > > > > > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > > > I am pleased to announce the availability of the second > beta of > > NumPy > > 1.5.0. This will be the first NumPy release to include > support for > > Python 3, as well as for Python 2.7. > > > > Please try this beta and report any problems on the NumPy > > mailing list. > > Especially with Python 3 testing will be very useful. On > Linux > > and OS X > > building from source should be straightforward, for > Windows a binary > > installer is provided. There is one important known issue > on Windows > > left, in fromfile and tofile (ticket 1583). > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > > > > > Enjoy, > > Ralf > > > > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and > 3.1 (32 > > and 64 bit) still reports many (> 200) warnings and three > known test > > failures/errors. Nothing serious, but it would be nice to > clean up > > before the final release. > > > > The warnings are of the type "Warning: invalid value > encountered in" > > for the functions reduce, fmax, fmin, logaddexp, maximum, > greater, > > less_equal, greater_equal, absolute, and others. I do not see > any of > > these warnings in the msvc9 builds of numpy 1.4.1. > > > > > > The warnings were accidentally turned off for earlier versions of > Numpy. > > I expect these warnings are related to nans and probably due to > problems > > with isnan or some such. Can you take a closer look? The fmax > function > > should be easy to check out. > > > > > > > > Chuck > > > > > Thanks for the hint. Warnings are issued in the test_umath test_*nan* > functions. The problem can be condensed to this statement: > > >>> numpy.array([numpy.nan]) > 0 > Warning: invalid value encountered in greater > array([False], dtype=bool) > > > When using msvc, ordered comparisons involving NaN raise an exception > [1], i.e. set the 'invalid' x87 status bit, which leads to the warning > being printed. I don't know if this violates IEEE 754 or C99 standards > but it does not happen with the gcc builds. Maybe > seterr(invalid='ignore') could be added to the test_*nan* functions? > > [1] http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx > > > OK, this does seem to be the standard. For instance > > The isless macro determines whether its first argument is less than its > second > argument. The value of isless(x, y) is always equal to (x) < (y); however, > unlike (x) < (y), isless(x, y) does not raise the ??invalid?? floating-point > exception when x and y are unordered. Yes, it is - but I cannot reproduce the INVALID FPU exception on Linux when using e.g. int a = (Nan > 1). I don't know what's up with that, as the glibc says it should, cheers, David From cgohlke at uci.edu Wed Aug 18 02:18:03 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 17 Aug 2010 23:18:03 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> Message-ID: <4C6B7B1B.5070704@uci.edu> On 8/17/2010 9:56 PM, Charles R Harris wrote: > > > On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke > wrote: > > > > On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke > > >> wrote: > > > > > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > > > I am pleased to announce the availability of the second > beta of > > NumPy > > 1.5.0. This will be the first NumPy release to include > support for > > Python 3, as well as for Python 2.7. > > > > Please try this beta and report any problems on the NumPy > > mailing list. > > Especially with Python 3 testing will be very useful. On Linux > > and OS X > > building from source should be straightforward, for > Windows a binary > > installer is provided. There is one important known issue > on Windows > > left, in fromfile and tofile (ticket 1583). > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > > > > > Enjoy, > > Ralf > > > > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 (32 > > and 64 bit) still reports many (> 200) warnings and three > known test > > failures/errors. Nothing serious, but it would be nice to clean up > > before the final release. > > > > The warnings are of the type "Warning: invalid value > encountered in" > > for the functions reduce, fmax, fmin, logaddexp, maximum, greater, > > less_equal, greater_equal, absolute, and others. I do not see > any of > > these warnings in the msvc9 builds of numpy 1.4.1. > > > > > > The warnings were accidentally turned off for earlier versions of > Numpy. > > I expect these warnings are related to nans and probably due to > problems > > with isnan or some such. Can you take a closer look? The fmax function > > should be easy to check out. > > > > > > > > Chuck > > > > > Thanks for the hint. Warnings are issued in the test_umath test_*nan* > functions. The problem can be condensed to this statement: > > > >> numpy.array([numpy.nan]) > 0 > Warning: invalid value encountered in greater > array([False], dtype=bool) > > > When using msvc, ordered comparisons involving NaN raise an exception > [1], i.e. set the 'invalid' x87 status bit, which leads to the warning > being printed. I don't know if this violates IEEE 754 or C99 standards > but it does not happen with the gcc builds. Maybe > seterr(invalid='ignore') could be added to the test_*nan* functions? > > [1] http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx > > > OK, this does seem to be the standard. For instance > > The isless macro determines whether its first argument is less than its > second > argument. The value of isless(x, y) is always equal to (x) < (y); however, > unlike (x) < (y), isless(x, y) does not raise the ??invalid?? floating-point > exception when x and y are unordered. > > There are other macros for the rest of the comparisons. Can you check if > MSVC has these macros available? > MSVC doesn't have these macros. It should not be too difficult to define them according to http://code.google.com/p/v8/source/browse/branches/bleeding_edge/src/platform-win32.cc int isless(double x, double y) { return isnan(x) || isnan(y) ? 0 : x < y; } int isgreater(double x, double y) { return isnan(x) || isnan(y) ? 0 : x > y; } -- Christoph From martin.raspaud at smhi.se Wed Aug 18 02:19:26 2010 From: martin.raspaud at smhi.se (Martin Raspaud) Date: Wed, 18 Aug 2010 08:19:26 +0200 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: References: Message-ID: <4C6B7B6E.50005@smhi.se> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Goldsmith skrev: > > > Ahh, cell arrays, they bring back memories. Makes you pine for a > dictionary, no? > > JDH > > > Not to mention writeline, readline, string concatenation using +, > English wording of loops, list comprehension, etc., etc., etc. - if > people only knew... Once upon a time, when my boss wanted me to use matlab, I found myself implementing a python interpreter in matlab... Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJMa3tuAAoJEBdvyODiyJI4uZMIALklumu4r37sRxmxA2Pk6eC3 BnDQZtWW5LRPFZWWlw1jnfaYIOo7zqYurjNJ7BVLWaUznFZC+PIS2NVQ3a3j6nZZ TcSOTg6VcmNNEGRNXjlIkVNLILO/Dm70uulCkxSNt2Q/7PqkDPxkg7P44AGzz23U euATu56AKqUbgn5tZdSqKgeXCmLmTwg8QiO+z86YI404YErvkAuJrO0qoG5Yiwl5 O6lLPW13llBQT/BRo+MMlWh4pjGYKQ0YN/OXku+OndyxN4az9Xdel5o/UD9cZkYQ TcCu422XEaSKRp0suRxEUjvfRHR0lNXwdxfvUfDtb7NzjNXp9uTXhLu3h2Iq32c= =p+J1 -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: martin_raspaud.vcf Type: text/x-vcard Size: 260 bytes Desc: not available URL: From eg at fft.be Wed Aug 18 03:13:12 2010 From: eg at fft.be (Eloi Gaudry) Date: Wed, 18 Aug 2010 09:13:12 +0200 Subject: [Numpy-discussion] Problems building NumPy with GotoBLAS In-Reply-To: <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> References: <33444.66.146.167.66.1282021089.squirrel@phoenix.liquidweb.com> <201008171343.44255.eg@fft.be> <61057.66.146.167.66.1282084788.squirrel@phoenix.liquidweb.com> Message-ID: <201008180913.12594.eg@fft.be> Peter, As mentionned, you need to replace the values of GOTODIR, LAPACKDIR and PYTHONDIR in this script with the ones matching your environment. Eloi On Wednesday 18 August 2010 00:39:48 ashford at whisperpc.com wrote: > Eloi, > > > please below a script that will build numpy using a relevant site.cfg for > > your configuration (you need to update GOTODIR and LAPACKDIR and > > PYTHONDIR): > > > > #copy site.cfg > > cp ./site.cfg PYTHONDIR/lib/python2.6/site-packages/numpy/distutils/. > > I believe this needs a $ prior to PYTHONDIR. > > I tried this, and the benchmark still came in at 8.5S. > > Any ideas? How long should the following benchmark take on a Core-i7 930, > with Atlas or Goto? > > time python -c "import numpy as N; a=N.random.randn(1000, 1000); N.dot(a, > a)" > > Thank you. > > Peter Ashford > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Eloi Gaudry Free Field Technologies Company Website: http://www.fft.be Company Phone: +32 10 487 959 From sturla at molden.no Wed Aug 18 03:20:41 2010 From: sturla at molden.no (Sturla Molden) Date: Wed, 18 Aug 2010 09:20:41 +0200 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: <4C6B7B6E.50005@smhi.se> References: <4C6B7B6E.50005@smhi.se> Message-ID: <8C0B2317-2A22-4828-99E8-AC6C0F77824F@molden.no> Den 18. aug. 2010 kl. 08.19 skrev Martin Raspaud : > Once upon a time, when my boss wanted me to use matlab, I found myself > implementing a python interpreter in matlab... > There are just two sane solutions for Matlab: Either embed CPyton in a MEX file, or use Matlab's JVM to run Jython ;) http://vader.cse.lehigh.edu/~perkins/pymex.html Sturla From robince at gmail.com Wed Aug 18 04:33:59 2010 From: robince at gmail.com (Robin) Date: Wed, 18 Aug 2010 09:33:59 +0100 Subject: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John Hunter) In-Reply-To: <8C0B2317-2A22-4828-99E8-AC6C0F77824F@molden.no> References: <4C6B7B6E.50005@smhi.se> <8C0B2317-2A22-4828-99E8-AC6C0F77824F@molden.no> Message-ID: On Wed, Aug 18, 2010 at 8:20 AM, Sturla Molden wrote: > Den 18. aug. 2010 kl. 08.19 skrev Martin Raspaud > : > >> Once upon a time, when my boss wanted me to use matlab, I found myself >> implementing a python interpreter in matlab... >> > > There are just two sane solutions for Matlab: Either embed CPyton in a > MEX file, or use Matlab's JVM to run Jython ;) > > http://vader.cse.lehigh.edu/~perkins/pymex.html Just thought I'd mention another one since this came up: http://github.com/kw/pymex This one works very nicely - it proxies any Python objects so you can use, should you want to, the Matlab IDE as a python interpreter, supports numpy arrays etc. Also cross-platform - I even got it to work with 64 bit matlab/python on windows (in a fork on github). > > > Sturla > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From macdonald at maths.ox.ac.uk Wed Aug 18 05:46:27 2010 From: macdonald at maths.ox.ac.uk (Colin Macdonald) Date: Wed, 18 Aug 2010 10:46:27 +0100 Subject: [Numpy-discussion] longdouble (float96) literals Message-ID: <4C6BABF3.9090201@maths.ox.ac.uk> How can I enter longdouble (float96) literals into my python/numpy programs? In C, I would postfix such numbers with 'L', but this gives a SyntaxError in python. The rest of my message is just two examples of what I'm talking about in case its not clear. Thanks, Colin ---------------------------------------------------------- Note: longdouble: machine epsilon of approx 1e-19 float64: mach eps approx 1e-16 For example, say I need a longdouble accurate 0.4 in my code: In [249]: f96 = numpy.longdouble In [250]: f96(0.4) * f96(10) - f96(4) Out[250]: 2.2204460492503130808e-16 # no, that converted 0.4 to a float64 first (and note the answer is machine epsilon for float64) In [251]: f96("0.4") * f96(10) - f96(4) Out[251]: 2.2204460492503130808e-16 # no In [252]: (f96(4)/f96(10)) * f96(10) - f96(4) Out[252]: 0.0 # this works (in this particular case). I also would've been happy with 1e-19 in the last case---of course I don't expect it to be exactly zero. Finally, here's another example involving pi, done without ipython: import numpy as np f96 = np.longdouble # of course this won't work because numpy.pi is a float64 mypi0 = f96(np.pi) print np.sin(mypi0) # in C, I would write: #mypi1 = 3.141592653589793238462643383279L mypi1 = 3.141592653589793238462643383279 print np.sin(mypi1) mypi2 = np.arctan2( f96(0), f96(-1) ) print np.sin(mypi2) OUTPUT: $ python f96_issue.py 1.22460635382e-16 1.22460635382e-16 -5.42101086243e-20 -- Colin Macdonald University Lecturer in Numerical Methodologies Tutorial Fellow at Oriel College University of Oxford From ndbecker2 at gmail.com Wed Aug 18 07:56:17 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 18 Aug 2010 07:56:17 -0400 Subject: [Numpy-discussion] clip applied to complex gives unexpected results Message-ID: The results of clip applied to complex values is surprising and not documented (AFAICT) From markbak at gmail.com Wed Aug 18 08:42:51 2010 From: markbak at gmail.com (Mark Bakker) Date: Wed, 18 Aug 2010 14:42:51 +0200 Subject: [Numpy-discussion] inconsistency in 10**(-2) or 10**array([-2]) Message-ID: Hello list, When I do 10**-2, Python nicely returns 0.02 But with numpy (version 1.3.0), I get zero: In [492]: 10**array([-2]) Out[492]: array([0]) Is this the intended behavior? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 18 08:43:35 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 06:43:35 -0600 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: <4C6BABF3.9090201@maths.ox.ac.uk> References: <4C6BABF3.9090201@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 3:46 AM, Colin Macdonald wrote: > How can I enter longdouble (float96) literals into my python/numpy > programs? In C, I would postfix such numbers with 'L', but this gives > a SyntaxError in python. > > The rest of my message is just two examples of what I'm talking about > in case its not clear. > > Thanks, > Colin > > ---------------------------------------------------------- > > Note: > longdouble: machine epsilon of approx 1e-19 > float64: mach eps approx 1e-16 > > For example, say I need a longdouble accurate 0.4 in my code: > > In [249]: f96 = numpy.longdouble > In [250]: f96(0.4) * f96(10) - f96(4) > Out[250]: 2.2204460492503130808e-16 > # no, that converted 0.4 to a float64 first (and note the answer is > machine epsilon for float64) > In [251]: f96("0.4") * f96(10) - f96(4) > Out[251]: 2.2204460492503130808e-16 > # no > In [252]: (f96(4)/f96(10)) * f96(10) - f96(4) > Out[252]: 0.0 > # this works (in this particular case). I also would've been happy > with 1e-19 in the last case---of course I don't expect it to be > exactly zero. > > Finally, here's another example involving pi, done without ipython: > > import numpy as np > f96 = np.longdouble > # of course this won't work because numpy.pi is a float64 > mypi0 = f96(np.pi) > print np.sin(mypi0) > # in C, I would write: > #mypi1 = 3.141592653589793238462643383279L > mypi1 = 3.141592653589793238462643383279 > print np.sin(mypi1) > mypi2 = np.arctan2( f96(0), f96(-1) ) > print np.sin(mypi2) > > OUTPUT: > $ python f96_issue.py > 1.22460635382e-16 > 1.22460635382e-16 > -5.42101086243e-20 > > I don't know how to do this without using strings and having numpy do it's own string conversions. IIRC, we currently rely on Python for this and Python doesn't support long doubles. You might open a ticket with a request for an enhancement as we should fix this at some point. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 18 08:50:10 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 06:50:10 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: <4C6B7B1B.5070704@uci.edu> References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> <4C6B7B1B.5070704@uci.edu> Message-ID: On Wed, Aug 18, 2010 at 12:18 AM, Christoph Gohlke wrote: > > > On 8/17/2010 9:56 PM, Charles R Harris wrote: > > > > > > On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke > > wrote: > > > > > > > > On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > > > > > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke < > cgohlke at uci.edu > > > > > >> wrote: > > > > > > > > > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > > > > > I am pleased to announce the availability of the second > > beta of > > > NumPy > > > 1.5.0. This will be the first NumPy release to include > > support for > > > Python 3, as well as for Python 2.7. > > > > > > Please try this beta and report any problems on the NumPy > > > mailing list. > > > Especially with Python 3 testing will be very useful. On > Linux > > > and OS X > > > building from source should be straightforward, for > > Windows a binary > > > installer is provided. There is one important known issue > > on Windows > > > left, in fromfile and tofile (ticket 1583). > > > > > > Binaries, sources and release notes can be found at > > > https://sourceforge.net/projects/numpy/files/ > > > > > > > > > Enjoy, > > > Ralf > > > > > > > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and 3.1 > (32 > > > and 64 bit) still reports many (> 200) warnings and three > > known test > > > failures/errors. Nothing serious, but it would be nice to > clean up > > > before the final release. > > > > > > The warnings are of the type "Warning: invalid value > > encountered in" > > > for the functions reduce, fmax, fmin, logaddexp, maximum, > greater, > > > less_equal, greater_equal, absolute, and others. I do not see > > any of > > > these warnings in the msvc9 builds of numpy 1.4.1. > > > > > > > > > The warnings were accidentally turned off for earlier versions of > > Numpy. > > > I expect these warnings are related to nans and probably due to > > problems > > > with isnan or some such. Can you take a closer look? The fmax > function > > > should be easy to check out. > > > > > > > > > > > > Chuck > > > > > > > > > Thanks for the hint. Warnings are issued in the test_umath test_*nan* > > functions. The problem can be condensed to this statement: > > > > > >> numpy.array([numpy.nan]) > 0 > > Warning: invalid value encountered in greater > > array([False], dtype=bool) > > > > > > When using msvc, ordered comparisons involving NaN raise an exception > > [1], i.e. set the 'invalid' x87 status bit, which leads to the > warning > > being printed. I don't know if this violates IEEE 754 or C99 > standards > > but it does not happen with the gcc builds. Maybe > > seterr(invalid='ignore') could be added to the test_*nan* functions? > > > > [1] > http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx > > > > > > OK, this does seem to be the standard. For instance > > > > The isless macro determines whether its first argument is less than its > > second > > argument. The value of isless(x, y) is always equal to (x) < (y); > however, > > unlike (x) < (y), isless(x, y) does not raise the ??invalid?? > floating-point > > exception when x and y are unordered. > > > > There are other macros for the rest of the comparisons. Can you check if > > MSVC has these macros available? > > > > MSVC doesn't have these macros. It should not be too difficult to define > them according to > > http://code.google.com/p/v8/source/browse/branches/bleeding_edge/src/platform-win32.cc > > int isless(double x, double y) { > return isnan(x) || isnan(y) ? 0 : x < y; > } > > > int isgreater(double x, double y) { > return isnan(x) || isnan(y) ? 0 : x > y; > } > > Looks like we should just rewrite the ufuncs. Having these checks will slow things down a bit but it is probably the easiest way to make things portable without doing a lot of complicated work to determine what the platform does. However, this means that numpy won't ever set exceptions when nans are encountered and this is a design decision we should make up front. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 18 09:00:28 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 07:00:28 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> <4C6B7B1B.5070704@uci.edu> Message-ID: On Wed, Aug 18, 2010 at 6:50 AM, Charles R Harris wrote: > > > On Wed, Aug 18, 2010 at 12:18 AM, Christoph Gohlke wrote: > >> >> >> On 8/17/2010 9:56 PM, Charles R Harris wrote: >> > >> > >> > On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke > > > wrote: >> > >> > >> > >> > On 8/17/2010 1:02 PM, Charles R Harris wrote: >> > > >> > > >> > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke < >> cgohlke at uci.edu >> > >> > > >> wrote: >> > > >> > > >> > > >> > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: >> > > >> > > I am pleased to announce the availability of the second >> > beta of >> > > NumPy >> > > 1.5.0. This will be the first NumPy release to include >> > support for >> > > Python 3, as well as for Python 2.7. >> > > >> > > Please try this beta and report any problems on the NumPy >> > > mailing list. >> > > Especially with Python 3 testing will be very useful. On >> Linux >> > > and OS X >> > > building from source should be straightforward, for >> > Windows a binary >> > > installer is provided. There is one important known issue >> > on Windows >> > > left, in fromfile and tofile (ticket 1583). >> > > >> > > Binaries, sources and release notes can be found at >> > > https://sourceforge.net/projects/numpy/files/ >> > > >> > > >> > > Enjoy, >> > > Ralf >> > > >> > > >> > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 and >> 3.1 (32 >> > > and 64 bit) still reports many (> 200) warnings and three >> > known test >> > > failures/errors. Nothing serious, but it would be nice to >> clean up >> > > before the final release. >> > > >> > > The warnings are of the type "Warning: invalid value >> > encountered in" >> > > for the functions reduce, fmax, fmin, logaddexp, maximum, >> greater, >> > > less_equal, greater_equal, absolute, and others. I do not see >> > any of >> > > these warnings in the msvc9 builds of numpy 1.4.1. >> > > >> > > >> > > The warnings were accidentally turned off for earlier versions of >> > Numpy. >> > > I expect these warnings are related to nans and probably due to >> > problems >> > > with isnan or some such. Can you take a closer look? The fmax >> function >> > > should be easy to check out. >> > > >> > > >> > > >> > > Chuck >> > > >> > >> > >> > Thanks for the hint. Warnings are issued in the test_umath >> test_*nan* >> > functions. The problem can be condensed to this statement: >> > >> > > >> numpy.array([numpy.nan]) > 0 >> > Warning: invalid value encountered in greater >> > array([False], dtype=bool) >> > >> > >> > When using msvc, ordered comparisons involving NaN raise an >> exception >> > [1], i.e. set the 'invalid' x87 status bit, which leads to the >> warning >> > being printed. I don't know if this violates IEEE 754 or C99 >> standards >> > but it does not happen with the gcc builds. Maybe >> > seterr(invalid='ignore') could be added to the test_*nan* functions? >> > >> > [1] >> http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx >> > >> > >> > OK, this does seem to be the standard. For instance >> > >> > The isless macro determines whether its first argument is less than its >> > second >> > argument. The value of isless(x, y) is always equal to (x) < (y); >> however, >> > unlike (x) < (y), isless(x, y) does not raise the ??invalid?? >> floating-point >> > exception when x and y are unordered. >> > >> > There are other macros for the rest of the comparisons. Can you check if >> > MSVC has these macros available? >> > >> >> MSVC doesn't have these macros. It should not be too difficult to define >> them according to >> >> http://code.google.com/p/v8/source/browse/branches/bleeding_edge/src/platform-win32.cc >> >> int isless(double x, double y) { >> return isnan(x) || isnan(y) ? 0 : x < y; >> } >> >> >> int isgreater(double x, double y) { >> return isnan(x) || isnan(y) ? 0 : x > y; >> } >> >> > Looks like we should just rewrite the ufuncs. Having these checks will slow > things down a bit but it is probably the easiest way to make things portable > without doing a lot of complicated work to determine what the platform does. > However, this means that numpy won't ever set exceptions when nans are > encountered and this is a design decision we should make up front. > > Thinking about it a bit more, I don't think we should fix the comparisons, rather we should fix the tests to ignore the warnings. OTOH, since we already have defined behaviors for max/min/fmax/fmin maybe we should fix those functions. Or maybe not, since warning on invalid values is orthogonal to what we do with them. Hmm... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmund.hjulstad at gmail.com Wed Aug 18 09:17:51 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Wed, 18 Aug 2010 16:17:51 +0300 Subject: [Numpy-discussion] Evaluating performance of f2py extensions with gprof, why spending time _gfortran_compare_string Message-ID: I am calling a few functions in a fortran library. All parameters are short (longest array of 20 elements), and I do three calls to the fortran library pr iteration. According to the python profiler (running the script as %run -p in ipython), all time is spent in the python extension. I built the extension with options -pg -O , ran a test script, and evaluated the output with gprof .py -b with the following output: Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls Ts/call Ts/call name 41.64 5.03 5.03 _gfortran_compare_string 27.40 8.34 3.31 rdxhmx_ 19.21 10.66 2.32 phimix_ 5.88 11.37 0.71 phifeq_ 2.32 11.65 0.28 phihmx_ 0.66 11.73 0.08 phiderv_ [...] and this call graph: Call graph granularity: each sample hit covers 4 byte(s) for 0.08% of 11.83 seconds index % time self children called name [1] 42.9 5.07 0.00 _gfortran_compare_string [1] ----------------------------------------------- What can this mean? Executing a simple test program, exercising many of the same methods, I don't see any _gfortran_compare_string in the output. Suggestions most welcome. -- ?smund Hjulstad, asmund at hjulstad.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto at dealmeida.net Wed Aug 18 09:31:03 2010 From: roberto at dealmeida.net (Roberto De Almeida) Date: Wed, 18 Aug 2010 10:31:03 -0300 Subject: [Numpy-discussion] inconsistency in 10**(-2) or 10**array([-2]) In-Reply-To: References: Message-ID: Hi, Mark. I think the problem is that array([-2]) is an array of integers, so the result is also an array of integers. It works fine with array([-2.0]). --Rob On Wed, Aug 18, 2010 at 9:42 AM, Mark Bakker wrote: > Hello list, > > When I do 10**-2, Python nicely returns 0.02 > > But with numpy (version 1.3.0), I get zero: > > In [492]: 10**array([-2]) > Out[492]: array([0]) > > Is this the intended behavior? > > Thanks, Mark > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From macdonald at maths.ox.ac.uk Wed Aug 18 09:37:14 2010 From: macdonald at maths.ox.ac.uk (Colin Macdonald) Date: Wed, 18 Aug 2010 14:37:14 +0100 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> Message-ID: <4C6BE20A.50201@maths.ox.ac.uk> On 08/18/10 13:43, Charles R Harris wrote: > > > On Wed, Aug 18, 2010 at 3:46 AM, Colin Macdonald > > wrote: > > How can I enter longdouble (float96) literals into my python/numpy > programs? In C, I would postfix such numbers with 'L', but this gives > a SyntaxError in python. > > I don't know how to do this without using strings and having numpy do > it's own string conversions. IIRC, we currently rely on Python for > this and Python doesn't support long doubles. You might open a ticket > with a request for an enhancement as we should fix this at some point. I'd be at least partially satisfied with doing a string conversion ;-) How would I do that? Is there something like strtolongdouble()? Or a sscanf function? I will try to remember to open a ticket for the issue. Perhaps the issue that numpy.longdouble("0.4") doesn't work properly a slightly separate bug... From markbak at gmail.com Wed Aug 18 09:40:27 2010 From: markbak at gmail.com (Mark Bakker) Date: Wed, 18 Aug 2010 15:40:27 +0200 Subject: [Numpy-discussion] inconsistency in 10**(-2) or 10**array([-2]) In-Reply-To: References: Message-ID: I understand why numpy does it, but even Python itself gives 10**-2 = 0.01. So I am wondering whether this is the intended behavior of numpy. I don't really think so, but I may be wrong. > Roberto wrote: > I think the problem is that array([-2]) is an array of integers, so the result is also an array of integers. It works fine with array([-2.0]). On Wed, Aug 18, 2010 at 2:42 PM, Mark Bakker wrote: > Hello list, > > When I do 10**-2, Python nicely returns 0.02 > > But with numpy (version 1.3.0), I get zero: > > In [492]: 10**array([-2]) > Out[492]: array([0]) > > Is this the intended behavior? > > Thanks, Mark > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 18 10:14:38 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 08:14:38 -0600 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: <4C6BE20A.50201@maths.ox.ac.uk> References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 7:37 AM, Colin Macdonald wrote: > On 08/18/10 13:43, Charles R Harris wrote: > > > > > > On Wed, Aug 18, 2010 at 3:46 AM, Colin Macdonald > > > wrote: > > > > How can I enter longdouble (float96) literals into my python/numpy > > programs? In C, I would postfix such numbers with 'L', but this > gives > > a SyntaxError in python. > > > > I don't know how to do this without using strings and having numpy do > > it's own string conversions. IIRC, we currently rely on Python for > > this and Python doesn't support long doubles. You might open a ticket > > with a request for an enhancement as we should fix this at some point. > > I'd be at least partially satisfied with doing a string conversion ;-) > How would I do that? Is there something like strtolongdouble()? Or a > sscanf function? > > If you are willing to work at the C-Level you can probably use sscanf with the proper format specifier. Or you might write a parser in python. Numpy just doesn't have support for this at the moment, at least not that I am aware of. However, the various constants supplied by numpy, pi and such, are full precision. > I will try to remember to open a ticket for the issue. > > Perhaps the issue that numpy.longdouble("0.4") doesn't work properly a > slightly separate bug... > I'm pretty sure it goes through a double and can't be fixed until there is long double support for string conversions. Unicode makes this even trickier as to use the C-library it needs to be converted to ascii first. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From macdonald at maths.ox.ac.uk Wed Aug 18 10:25:47 2010 From: macdonald at maths.ox.ac.uk (Colin Macdonald) Date: Wed, 18 Aug 2010 15:25:47 +0100 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> Message-ID: <4C6BED6B.5080204@maths.ox.ac.uk> On 08/18/10 15:14, Charles R Harris wrote: > However, the various constants supplied by numpy, pi and such, are > full precision. no, they are not. My example demonstrated that numpy.pi is only double precision. Thanks for your help so far. Colin From charlesr.harris at gmail.com Wed Aug 18 10:36:15 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 08:36:15 -0600 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: <4C6BED6B.5080204@maths.ox.ac.uk> References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 8:25 AM, Colin Macdonald wrote: > On 08/18/10 15:14, Charles R Harris wrote: > > However, the various constants supplied by numpy, pi and such, are > > full precision. > > no, they are not. My example demonstrated that numpy.pi is only > double precision. > > Hmm, the full precision values are available internally but it looks like they aren't available to the public. I wonder what the easiest way to provide them would be? Maybe they should be long double types by default? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 18 10:52:54 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 18 Aug 2010 10:52:54 -0400 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 10:36 AM, Charles R Harris wrote: > > > On Wed, Aug 18, 2010 at 8:25 AM, Colin Macdonald > wrote: >> >> On 08/18/10 15:14, Charles R Harris wrote: >> > However, the various constants supplied by numpy, pi and such, are >> > full precision. >> >> no, they are not. ?My example demonstrated that numpy.pi is only >> double precision. >> > > Hmm, the full precision values are available internally but it looks like > they aren't available to the public. I wonder what the easiest way to > provide them would be? Maybe they should be long double types by default? playing with some examples, I don't seem to be able to do anything with longdouble on win32, py2.5 >>> np.array([3141592653589793238L], np.int64).astype(np.longdouble)[0] 3141592653589793300.0 >>> np.array([3141592653589793238L], np.int64).astype(float)[0] 3.1415926535897933e+018 >>> 1./np.array([np.pi],np.longdouble)[0] - 1/np.pi 0.0 >>> 1./np.array([np.pi],np.longdouble)[0] 0.31830988618379069 and it doesn't look like it's the print precision >>> 1./np.array([np.pi],np.longdouble)[0]*1e18 318309886183790720.0 >>> 1./np.array([np.pi],float)[0]*1e18 3.1830988618379072e+017 type conversion and calculations seem to go through float Josef > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Wed Aug 18 10:54:59 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 18 Aug 2010 16:54:59 +0200 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: I don't think there is longdouble on Windows, is there? Matthieu 2010/8/18 : > On Wed, Aug 18, 2010 at 10:36 AM, Charles R Harris > wrote: >> >> >> On Wed, Aug 18, 2010 at 8:25 AM, Colin Macdonald >> wrote: >>> >>> On 08/18/10 15:14, Charles R Harris wrote: >>> > However, the various constants supplied by numpy, pi and such, are >>> > full precision. >>> >>> no, they are not. ?My example demonstrated that numpy.pi is only >>> double precision. >>> >> >> Hmm, the full precision values are available internally but it looks like >> they aren't available to the public. I wonder what the easiest way to >> provide them would be? Maybe they should be long double types by default? > > playing with some examples, I don't seem to be able to do anything > with longdouble on win32, py2.5 > >>>> np.array([3141592653589793238L], np.int64).astype(np.longdouble)[0] > 3141592653589793300.0 >>>> np.array([3141592653589793238L], np.int64).astype(float)[0] > 3.1415926535897933e+018 >>>> 1./np.array([np.pi],np.longdouble)[0] - 1/np.pi > 0.0 >>>> 1./np.array([np.pi],np.longdouble)[0] > 0.31830988618379069 > > and it doesn't look like it's the print precision >>>> 1./np.array([np.pi],np.longdouble)[0]*1e18 > 318309886183790720.0 >>>> 1./np.array([np.pi],float)[0]*1e18 > 3.1830988618379072e+017 > > > type conversion and calculations seem to go through float > > Josef > >> >> Chuck >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From charlesr.harris at gmail.com Wed Aug 18 10:58:47 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 08:58:47 -0600 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 8:54 AM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > I don't think there is longdouble on Windows, is there? > > The MSVC compilers don't support extended precision, or rather, long doubles are the same as doubles. I don't know what other compilers on windows do. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 18 11:07:01 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 18 Aug 2010 11:07:01 -0400 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 10:58 AM, Charles R Harris wrote: > > > On Wed, Aug 18, 2010 at 8:54 AM, Matthieu Brucher > wrote: >> >> I don't think there is longdouble on Windows, is there? >>> np.longdouble >>> np.longdouble(5).itemsize 12 >>> np.double(5).itemsize 8 >> > > The MSVC compilers don't support extended precision, or rather, long doubles > are the same as doubles. I don't know what other compilers on windows do. I have the mingw version, gcc3.5 or so So to me it looks like float96 is there, but not used in calculations. (I never tried to use it for anything before.) Josef > > > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From cournape at gmail.com Wed Aug 18 11:07:35 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 19 Aug 2010 00:07:35 +0900 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 11:52 PM, wrote: > On Wed, Aug 18, 2010 at 10:36 AM, Charles R Harris > wrote: >> >> >> On Wed, Aug 18, 2010 at 8:25 AM, Colin Macdonald >> wrote: >>> >>> On 08/18/10 15:14, Charles R Harris wrote: >>> > However, the various constants supplied by numpy, pi and such, are >>> > full precision. >>> >>> no, they are not. ?My example demonstrated that numpy.pi is only >>> double precision. >>> >> >> Hmm, the full precision values are available internally but it looks like >> they aren't available to the public. I wonder what the easiest way to >> provide them would be? Maybe they should be long double types by default? > > playing with some examples, I don't seem to be able to do anything > with longdouble on win32, py2.5 For all practical purposes, double == long double on windows: although some compilers support the intel long format or even quadd precision, the MSVC runtime does not (so you cannot print, scan, etc... making them useless). cheers, David > >>>> np.array([3141592653589793238L], np.int64).astype(np.longdouble)[0] > 3141592653589793300.0 >>>> np.array([3141592653589793238L], np.int64).astype(float)[0] > 3.1415926535897933e+018 >>>> 1./np.array([np.pi],np.longdouble)[0] - 1/np.pi > 0.0 >>>> 1./np.array([np.pi],np.longdouble)[0] > 0.31830988618379069 > > and it doesn't look like it's the print precision >>>> 1./np.array([np.pi],np.longdouble)[0]*1e18 > 318309886183790720.0 >>>> 1./np.array([np.pi],float)[0]*1e18 > 3.1830988618379072e+017 > > > type conversion and calculations seem to go through float > > Josef > >> >> Chuck >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Wed Aug 18 11:09:33 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 18 Aug 2010 11:09:33 -0400 Subject: [Numpy-discussion] longdouble (float96) literals In-Reply-To: References: <4C6BABF3.9090201@maths.ox.ac.uk> <4C6BE20A.50201@maths.ox.ac.uk> <4C6BED6B.5080204@maths.ox.ac.uk> Message-ID: On Wed, Aug 18, 2010 at 11:07 AM, David Cournapeau wrote: > On Wed, Aug 18, 2010 at 11:52 PM, ? wrote: >> On Wed, Aug 18, 2010 at 10:36 AM, Charles R Harris >> wrote: >>> >>> >>> On Wed, Aug 18, 2010 at 8:25 AM, Colin Macdonald >>> wrote: >>>> >>>> On 08/18/10 15:14, Charles R Harris wrote: >>>> > However, the various constants supplied by numpy, pi and such, are >>>> > full precision. >>>> >>>> no, they are not. ?My example demonstrated that numpy.pi is only >>>> double precision. >>>> >>> >>> Hmm, the full precision values are available internally but it looks like >>> they aren't available to the public. I wonder what the easiest way to >>> provide them would be? Maybe they should be long double types by default? >> >> playing with some examples, I don't seem to be able to do anything >> with longdouble on win32, py2.5 > > For all practical purposes, double == long double on windows: although > some compilers support the intel long format or even quadd precision, > the MSVC runtime does not (so you cannot print, scan, etc... making > them useless). Thanks for the info, I can stop playing Josef > > cheers, > > David >> >>>>> np.array([3141592653589793238L], np.int64).astype(np.longdouble)[0] >> 3141592653589793300.0 >>>>> np.array([3141592653589793238L], np.int64).astype(float)[0] >> 3.1415926535897933e+018 >>>>> 1./np.array([np.pi],np.longdouble)[0] - 1/np.pi >> 0.0 >>>>> 1./np.array([np.pi],np.longdouble)[0] >> 0.31830988618379069 >> >> and it doesn't look like it's the print precision >>>>> 1./np.array([np.pi],np.longdouble)[0]*1e18 >> 318309886183790720.0 >>>>> 1./np.array([np.pi],float)[0]*1e18 >> 3.1830988618379072e+017 >> >> >> type conversion and calculations seem to go through float >> >> Josef >> >>> >>> Chuck >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From bsouthey at gmail.com Wed Aug 18 12:02:33 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 18 Aug 2010 11:02:33 -0500 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: References: <4C6AF47D.6070900@gmail.com> Message-ID: <4C6C0419.10802@gmail.com> On 08/17/2010 04:34 PM, Charles R Harris wrote: > > > On Tue, Aug 17, 2010 at 2:43 PM, Bruce Southey > wrote: > > On 08/16/2010 10:00 PM, Charles R Harris wrote: > > Hi All, > > > > I just added support for Legendre polynomials to numpy and I > think the > > numpy.polynomial name space is getting a bit crowded. Since most of > > the current functions in that namespace are just used to > implement the > > Polynomial, Chebyshev, and Legendre classes I'm thinking of only > > importing those classes by default and leaving the other > functions to > > explicit imports. Of course I will have to fix the examples and > maybe > > some other users will be inconvenienced by the change. But with > 2.0.0 > > in the works this might be a good time to do this. Thoughts? > > > > Chuck > While I don't know a lot about this so things will be easily off base. > > In looking at the names, I did see many names that seem identical > except > that these work just with one type of polynomial. > > Obviously cheb2poly and poly2cheb are the conversion between the > polynomial and Chebyshev types - similarly leg2poly and poly2leg > for the > polynomial and Legendre classes. But none between Chebyshev and > Legendre > classes. Would it make more sense to create a single conversion > function > to change one type into another instead of the current 6 > possibilities? > > > The class types can be converted to each other, with an optional > change of domain, using the convert method, i.e., if p is an instance > of Legendre > > p.convert(kind=Chebyshev) > > will do the conversion to a Chebyshev series.. The classes don't > actually use the *2* functions, oddly enough ;) > > > > Similarily there are obviously a very similar functions that just work > with one polynomial type so the functionality is duplicated across > each > class that could be a single function each: > chebadd legadd polyadd > chebder legder polyder > chebdiv legdiv polydiv > chebdomain legdomain polydomain > chebfit legfit polyfit > chebfromroots legfromroots polyfromroots > chebint legint polyint > chebline legline polyline > chebmul legmul polymul > chebmulx legmulx polymulx > chebone legone polyone > chebroots legroots polyroots > chebsub legsub polysub > chebtrim legtrim polytrim > chebval legval polyval > chebvander legvander polyvander > chebx legx polyx > chebzero legzero polyzero > > However, I doubt that is worth the work if the overall amount of > code is > not reduced. For example, if you create a overall function that just > calls the appropriate add function for that type of polynomial > then I do > not see any advantage in doing so just to reduce the namespace. > If you can argue that is very beneficial to the user of polynomial > functions then that could put a different spin on doing that. > > While I would have to check more carefully (as I don't have time now), > aren't chebadd, legadd and polyadd essentially the same function? > That is, can you send a Legendre polynomial to the same Chebysnev > function and get the same answer back? > If so then these functions should be collapsed into one for numpy 2.0. > > > Yeah, the add and subtract functions are all the same along with the > *trim functions. These things are all accessable through the classes > ustng +/- and the trim and truncate methods. Which is why for normal > work I think the classes are the way to go, the functions are just for > implementing the classes and available in case someone wants to roll > their own. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Thanks, Now that I understand things a little better, I would agree with your original suggestion is to only import the class and leave the other functions as explicit imports. But I would go further and say that these other functions should be depreciated and removed especially if polynomial users consider classes should be used. If someone wants to roll their own, then they probably know enough to do so without needing these functions. Also, one of the things I noticed is that there is no checks on these other functions for the type of polynomial such that any input other than the correct input could be used. Sure user beware but at least with a class the correct input should be used. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 18 12:36:22 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 10:36:22 -0600 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: <4C6C0419.10802@gmail.com> References: <4C6AF47D.6070900@gmail.com> <4C6C0419.10802@gmail.com> Message-ID: On Wed, Aug 18, 2010 at 10:02 AM, Bruce Southey wrote: > On 08/17/2010 04:34 PM, Charles R Harris wrote: > > > > On Tue, Aug 17, 2010 at 2:43 PM, Bruce Southey wrote: > >> On 08/16/2010 10:00 PM, Charles R Harris wrote: >> > Hi All, >> > >> > I just added support for Legendre polynomials to numpy and I think the >> > numpy.polynomial name space is getting a bit crowded. Since most of >> > the current functions in that namespace are just used to implement the >> > Polynomial, Chebyshev, and Legendre classes I'm thinking of only >> > importing those classes by default and leaving the other functions to >> > explicit imports. Of course I will have to fix the examples and maybe >> > some other users will be inconvenienced by the change. But with 2.0.0 >> > in the works this might be a good time to do this. Thoughts? >> > >> > Chuck >> While I don't know a lot about this so things will be easily off base. >> >> In looking at the names, I did see many names that seem identical except >> that these work just with one type of polynomial. >> >> Obviously cheb2poly and poly2cheb are the conversion between the >> polynomial and Chebyshev types - similarly leg2poly and poly2leg for the >> polynomial and Legendre classes. But none between Chebyshev and Legendre >> classes. Would it make more sense to create a single conversion function >> to change one type into another instead of the current 6 possibilities? >> >> > The class types can be converted to each other, with an optional change of > domain, using the convert method, i.e., if p is an instance of Legendre > > p.convert(kind=Chebyshev) > > will do the conversion to a Chebyshev series.. The classes don't actually > use the *2* functions, oddly enough ;) > > > > >> Similarily there are obviously a very similar functions that just work >> with one polynomial type so the functionality is duplicated across each >> class that could be a single function each: >> chebadd legadd polyadd >> chebder legder polyder >> chebdiv legdiv polydiv >> chebdomain legdomain polydomain >> chebfit legfit polyfit >> chebfromroots legfromroots polyfromroots >> chebint legint polyint >> chebline legline polyline >> chebmul legmul polymul >> chebmulx legmulx polymulx >> chebone legone polyone >> chebroots legroots polyroots >> chebsub legsub polysub >> chebtrim legtrim polytrim >> chebval legval polyval >> chebvander legvander polyvander >> chebx legx polyx >> chebzero legzero polyzero >> >> However, I doubt that is worth the work if the overall amount of code is >> not reduced. For example, if you create a overall function that just >> calls the appropriate add function for that type of polynomial then I do >> not see any advantage in doing so just to reduce the namespace. >> If you can argue that is very beneficial to the user of polynomial >> functions then that could put a different spin on doing that. >> >> While I would have to check more carefully (as I don't have time now), >> aren't chebadd, legadd and polyadd essentially the same function? >> That is, can you send a Legendre polynomial to the same Chebysnev >> function and get the same answer back? >> If so then these functions should be collapsed into one for numpy 2.0. >> >> > Yeah, the add and subtract functions are all the same along with the *trim > functions. These things are all accessable through the classes ustng +/- and > the trim and truncate methods. Which is why for normal work I think the > classes are the way to go, the functions are just for implementing the > classes and available in case someone wants to roll their own. > > The various classes are generated from a single string template and need the functions. The classes implement a common interface, the functions do what is specific to the various types of polynomial. In general it is a good idea to keep the specific bits out of classes since designing *the* universal class is hard and anyone who wants to just borrow a bit of code will end up cursing the SOB who buried the good stuff in a class, creating all sorts of inconvenient dependencies. That's my experience, anyway. I also wanted to keep open the possibility of using cython to speed up specific small bits of the code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 18 13:27:36 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 18 Aug 2010 13:27:36 -0400 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: References: <4C6AF47D.6070900@gmail.com> <4C6C0419.10802@gmail.com> Message-ID: On Wed, Aug 18, 2010 at 12:36 PM, Charles R Harris wrote: > > > On Wed, Aug 18, 2010 at 10:02 AM, Bruce Southey wrote: >> >> On 08/17/2010 04:34 PM, Charles R Harris wrote: >> >> On Tue, Aug 17, 2010 at 2:43 PM, Bruce Southey wrote: >>> >>> ?On 08/16/2010 10:00 PM, Charles R Harris wrote: >>> > Hi All, >>> > >>> > I just added support for Legendre polynomials to numpy and I think the >>> > numpy.polynomial name space is getting a bit crowded. Since most of >>> > the current functions in that namespace are just used to implement the >>> > Polynomial, Chebyshev, and Legendre classes I'm thinking of only >>> > importing those classes by default and leaving the other functions to >>> > explicit imports. Of course I will have to fix the examples and maybe >>> > some other users will be inconvenienced by the change. But with 2.0.0 >>> > in the works this might be a good time to do this. Thoughts? >>> > >>> > Chuck >>> While I don't know a lot about this so things will be easily off base. >>> >>> In looking at the names, I did see many names that seem identical except >>> that these work just with one type of polynomial. >>> >>> Obviously cheb2poly and poly2cheb are the conversion between the >>> polynomial and Chebyshev types - similarly leg2poly and poly2leg for the >>> polynomial and Legendre classes. But none between Chebyshev and Legendre >>> classes. Would it make more sense to create a single conversion function >>> to change one type into another instead of the current 6 possibilities? >>> >> >> The class types can be converted to each other, with an optional change of >> domain, using the convert method, i.e., if p is an instance of Legendre >> >> p.convert(kind=Chebyshev) >> >> will do the conversion to a Chebyshev series.. The classes don't actually >> use the *2* functions, oddly enough ;) >> >> >> >>> >>> Similarily there are obviously a very similar functions that just work >>> with one polynomial type so the functionality is duplicated across each >>> class that could be a single function each: >>> chebadd ? ?legadd ? ?polyadd >>> chebder ? ?legder ? ?polyder >>> chebdiv ? ?legdiv ? ?polydiv >>> chebdomain ? ?legdomain ? ?polydomain >>> chebfit ? ?legfit ? ?polyfit >>> chebfromroots ? ?legfromroots ? ?polyfromroots >>> chebint ? ?legint ? ?polyint >>> chebline ? ?legline ? ?polyline >>> chebmul ? ?legmul ? ?polymul >>> chebmulx ? ?legmulx ? ?polymulx >>> chebone ? ?legone ? ?polyone >>> chebroots ? ?legroots ? ?polyroots >>> chebsub ? ?legsub ? ?polysub >>> chebtrim ? ?legtrim ? ?polytrim >>> chebval ? ?legval ? ?polyval >>> chebvander ? ?legvander ? ?polyvander >>> chebx ? ?legx ? ?polyx >>> chebzero ? ?legzero ? ?polyzero >>> >>> However, I doubt that is worth the work if the overall amount of code is >>> not reduced. For example, if you create a overall function that just >>> calls the appropriate add function for that type of polynomial then I do >>> not see any advantage in doing so just to reduce the namespace. >>> If you can argue that is very beneficial to the user of polynomial >>> functions then that could put a different spin on doing that. >>> >>> While I would have to check more carefully (as I don't have time now), >>> aren't chebadd, legadd and polyadd essentially the same function? >>> That is, can you send a Legendre polynomial to the same Chebysnev >>> function and get the same answer back? >>> If so then these functions should be collapsed into one for numpy 2.0. >>> >> >> Yeah, the add and subtract functions are all the same along with the *trim >> functions. These things are all accessable through the classes ustng +/- and >> the trim and truncate methods. Which is why for normal work I think the >> classes are the way to go, the functions are just for implementing the >> classes and available in case someone wants to roll their own. >> > > The various classes are generated from a single string template and need the > functions. The classes implement a common interface, the functions do what > is specific to the various types of polynomial. In general it is a good idea > to keep the specific bits out of classes since designing *the* universal > class is hard and anyone who wants to just borrow a bit of code will end up > cursing the SOB who buried the good stuff in a class, creating all sorts of > inconvenient dependencies. That's my experience, anyway. I also wanted to > keep open the possibility of using cython to speed up specific small bits of > the code. I also like internal code that can be borrowed. One possible idea if you keep extending polynomial and the number of modules and unique names is to import the extra functions into a common module but not into the main namespace. e.g. poly.py -- from polynomial import * from chebyshev import * from polyutils import * --- and import only the classes into the main namespace e.g. from np.polynomial.poly import chebvander, chebfit, polyfit (chebvander might be nicer than chebfit, because I can also calculate the covariance matrix :) (haven't tried it yet) Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Wed Aug 18 13:38:34 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Aug 2010 11:38:34 -0600 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: References: <4C6AF47D.6070900@gmail.com> <4C6C0419.10802@gmail.com> Message-ID: On Wed, Aug 18, 2010 at 11:27 AM, wrote: > On Wed, Aug 18, 2010 at 12:36 PM, Charles R Harris > wrote: > > > > > > On Wed, Aug 18, 2010 at 10:02 AM, Bruce Southey > wrote: > >> > >> On 08/17/2010 04:34 PM, Charles R Harris wrote: > >> > >> On Tue, Aug 17, 2010 at 2:43 PM, Bruce Southey > wrote: > >>> > >>> On 08/16/2010 10:00 PM, Charles R Harris wrote: > >>> > Hi All, > >>> > > >>> > I just added support for Legendre polynomials to numpy and I think > the > >>> > numpy.polynomial name space is getting a bit crowded. Since most of > >>> > the current functions in that namespace are just used to implement > the > >>> > Polynomial, Chebyshev, and Legendre classes I'm thinking of only > >>> > importing those classes by default and leaving the other functions to > >>> > explicit imports. Of course I will have to fix the examples and maybe > >>> > some other users will be inconvenienced by the change. But with 2.0.0 > >>> > in the works this might be a good time to do this. Thoughts? > >>> > > >>> > Chuck > >>> While I don't know a lot about this so things will be easily off base. > >>> > >>> In looking at the names, I did see many names that seem identical > except > >>> that these work just with one type of polynomial. > >>> > >>> Obviously cheb2poly and poly2cheb are the conversion between the > >>> polynomial and Chebyshev types - similarly leg2poly and poly2leg for > the > >>> polynomial and Legendre classes. But none between Chebyshev and > Legendre > >>> classes. Would it make more sense to create a single conversion > function > >>> to change one type into another instead of the current 6 possibilities? > >>> > >> > >> The class types can be converted to each other, with an optional change > of > >> domain, using the convert method, i.e., if p is an instance of Legendre > >> > >> p.convert(kind=Chebyshev) > >> > >> will do the conversion to a Chebyshev series.. The classes don't > actually > >> use the *2* functions, oddly enough ;) > >> > >> > >> > >>> > >>> Similarily there are obviously a very similar functions that just work > >>> with one polynomial type so the functionality is duplicated across each > >>> class that could be a single function each: > >>> chebadd legadd polyadd > >>> chebder legder polyder > >>> chebdiv legdiv polydiv > >>> chebdomain legdomain polydomain > >>> chebfit legfit polyfit > >>> chebfromroots legfromroots polyfromroots > >>> chebint legint polyint > >>> chebline legline polyline > >>> chebmul legmul polymul > >>> chebmulx legmulx polymulx > >>> chebone legone polyone > >>> chebroots legroots polyroots > >>> chebsub legsub polysub > >>> chebtrim legtrim polytrim > >>> chebval legval polyval > >>> chebvander legvander polyvander > >>> chebx legx polyx > >>> chebzero legzero polyzero > >>> > >>> However, I doubt that is worth the work if the overall amount of code > is > >>> not reduced. For example, if you create a overall function that just > >>> calls the appropriate add function for that type of polynomial then I > do > >>> not see any advantage in doing so just to reduce the namespace. > >>> If you can argue that is very beneficial to the user of polynomial > >>> functions then that could put a different spin on doing that. > >>> > >>> While I would have to check more carefully (as I don't have time now), > >>> aren't chebadd, legadd and polyadd essentially the same function? > >>> That is, can you send a Legendre polynomial to the same Chebysnev > >>> function and get the same answer back? > >>> If so then these functions should be collapsed into one for numpy 2.0. > >>> > >> > >> Yeah, the add and subtract functions are all the same along with the > *trim > >> functions. These things are all accessable through the classes ustng +/- > and > >> the trim and truncate methods. Which is why for normal work I think the > >> classes are the way to go, the functions are just for implementing the > >> classes and available in case someone wants to roll their own. > >> > > > > The various classes are generated from a single string template and need > the > > functions. The classes implement a common interface, the functions do > what > > is specific to the various types of polynomial. In general it is a good > idea > > to keep the specific bits out of classes since designing *the* universal > > class is hard and anyone who wants to just borrow a bit of code will end > up > > cursing the SOB who buried the good stuff in a class, creating all sorts > of > > inconvenient dependencies. That's my experience, anyway. I also wanted to > > keep open the possibility of using cython to speed up specific small bits > of > > the code. > > I also like internal code that can be borrowed. > > One possible idea if you keep extending polynomial and the number of > modules and unique names is to import the extra functions into a > common module but not into the main namespace. > > I'm sorta of thinking about adding Hermite and Laguerre polynomials. I can't quite figure how to include domains like [-inf, inf] into the way things are currently done, at least in a way that seems intuitive. e.g. poly.py > -- > from polynomial import * > from chebyshev import * > from polyutils import * > --- > > and import only the classes into the main namespace > > e.g. > from np.polynomial.poly import chebvander, chebfit, polyfit > > That's a thought. > (chebvander might be nicer than chebfit, because I can also calculate > the covariance matrix :) > (haven't tried it yet) > > I'd like to make getting the covariance an option but that means bypassing the version of lstsq that is in linalg. Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Aug 18 13:42:23 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 18 Aug 2010 13:42:23 -0400 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: References: <4C6AF47D.6070900@gmail.com> <4C6C0419.10802@gmail.com> Message-ID: On Wed, Aug 18, 2010 at 1:27 PM, wrote: > On Wed, Aug 18, 2010 at 12:36 PM, Charles R Harris > wrote: >> >> >> On Wed, Aug 18, 2010 at 10:02 AM, Bruce Southey wrote: >>> >>> On 08/17/2010 04:34 PM, Charles R Harris wrote: >>> >>> On Tue, Aug 17, 2010 at 2:43 PM, Bruce Southey wrote: >>>> >>>> ?On 08/16/2010 10:00 PM, Charles R Harris wrote: >>>> > Hi All, >>>> > >>>> > I just added support for Legendre polynomials to numpy and I think the >>>> > numpy.polynomial name space is getting a bit crowded. Since most of >>>> > the current functions in that namespace are just used to implement the >>>> > Polynomial, Chebyshev, and Legendre classes I'm thinking of only >>>> > importing those classes by default and leaving the other functions to >>>> > explicit imports. Of course I will have to fix the examples and maybe >>>> > some other users will be inconvenienced by the change. But with 2.0.0 >>>> > in the works this might be a good time to do this. Thoughts? >>>> > >>>> > Chuck >>>> While I don't know a lot about this so things will be easily off base. >>>> >>>> In looking at the names, I did see many names that seem identical except >>>> that these work just with one type of polynomial. >>>> >>>> Obviously cheb2poly and poly2cheb are the conversion between the >>>> polynomial and Chebyshev types - similarly leg2poly and poly2leg for the >>>> polynomial and Legendre classes. But none between Chebyshev and Legendre >>>> classes. Would it make more sense to create a single conversion function >>>> to change one type into another instead of the current 6 possibilities? >>>> >>> >>> The class types can be converted to each other, with an optional change of >>> domain, using the convert method, i.e., if p is an instance of Legendre >>> >>> p.convert(kind=Chebyshev) >>> >>> will do the conversion to a Chebyshev series.. The classes don't actually >>> use the *2* functions, oddly enough ;) >>> >>> >>> >>>> >>>> Similarily there are obviously a very similar functions that just work >>>> with one polynomial type so the functionality is duplicated across each >>>> class that could be a single function each: >>>> chebadd ? ?legadd ? ?polyadd >>>> chebder ? ?legder ? ?polyder >>>> chebdiv ? ?legdiv ? ?polydiv >>>> chebdomain ? ?legdomain ? ?polydomain >>>> chebfit ? ?legfit ? ?polyfit >>>> chebfromroots ? ?legfromroots ? ?polyfromroots >>>> chebint ? ?legint ? ?polyint >>>> chebline ? ?legline ? ?polyline >>>> chebmul ? ?legmul ? ?polymul >>>> chebmulx ? ?legmulx ? ?polymulx >>>> chebone ? ?legone ? ?polyone >>>> chebroots ? ?legroots ? ?polyroots >>>> chebsub ? ?legsub ? ?polysub >>>> chebtrim ? ?legtrim ? ?polytrim >>>> chebval ? ?legval ? ?polyval >>>> chebvander ? ?legvander ? ?polyvander >>>> chebx ? ?legx ? ?polyx >>>> chebzero ? ?legzero ? ?polyzero >>>> >>>> However, I doubt that is worth the work if the overall amount of code is >>>> not reduced. For example, if you create a overall function that just >>>> calls the appropriate add function for that type of polynomial then I do >>>> not see any advantage in doing so just to reduce the namespace. >>>> If you can argue that is very beneficial to the user of polynomial >>>> functions then that could put a different spin on doing that. >>>> >>>> While I would have to check more carefully (as I don't have time now), >>>> aren't chebadd, legadd and polyadd essentially the same function? >>>> That is, can you send a Legendre polynomial to the same Chebysnev >>>> function and get the same answer back? >>>> If so then these functions should be collapsed into one for numpy 2.0. >>>> >>> >>> Yeah, the add and subtract functions are all the same along with the *trim >>> functions. These things are all accessable through the classes ustng +/- and >>> the trim and truncate methods. Which is why for normal work I think the >>> classes are the way to go, the functions are just for implementing the >>> classes and available in case someone wants to roll their own. >>> >> >> The various classes are generated from a single string template and need the >> functions. The classes implement a common interface, the functions do what >> is specific to the various types of polynomial. In general it is a good idea >> to keep the specific bits out of classes since designing *the* universal >> class is hard and anyone who wants to just borrow a bit of code will end up >> cursing the SOB who buried the good stuff in a class, creating all sorts of >> inconvenient dependencies. That's my experience, anyway. I also wanted to >> keep open the possibility of using cython to speed up specific small bits of >> the code. > > I also like internal code that can be borrowed. > > One possible idea if you keep extending polynomial and the number of > modules and unique names is to import the extra functions into a > common module but not into the main namespace. > > e.g. poly.py > -- > from polynomial import * > from chebyshev import * > from polyutils import * > --- > > and import only the classes into the main namespace > > e.g. > from np.polynomial.poly import chebvander, chebfit, polyfit > > (chebvander might be nicer than chebfit, because I can also calculate > the covariance matrix :) > (haven't tried it yet) just checking >>> from numpy.polynomial import chebvander, chebfit >>> x = chebvander(np.arange(20),3) >>> y = x.sum(1) + 0.1*np.random.randn(x.shape[0]) >>> chebfit(np.arange(20), y, 3) array([ 1.06500672, 0.96512466, 1.00266968, 0.9999477 ]) >>> from scikits.statsmodels import OLS >>> res = OLS(y, x).fit() >>> res.params array([ 1.06500672, 0.96512466, 1.00266968, 0.9999477 ]) >>> res.bse array([ 1.14104979e-01, 5.24951183e-02, 3.25000334e-03, 5.61553371e-05]) >>> res.t() array([ 9.33356923e+00, 1.83850363e+01, 3.08513430e+02, 1.78068150e+04]) Josef > > Josef > >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From jbauer-news at web.de Wed Aug 18 14:07:48 2010 From: jbauer-news at web.de (Johann Bauer) Date: Wed, 18 Aug 2010 11:07:48 -0700 Subject: [Numpy-discussion] index of the first element fulfilling a condition? Message-ID: <4C6C2174.7060905@web.de> Hi, is there a good method to find the index of the first element in a 1D array fulfilling a condition? The following does the job >>> import numpy >>> a = numpy.array([1,5,78,3,12,4]) >>> numpy.where( a>10 )[0][0] 2 but it first compares the entire array and then selects the first index. Is there a way (especially for big arrays) to stop the comparison after the first hit? Thanks for any help, Johann From dalcinl at gmail.com Wed Aug 18 15:12:35 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 16:12:35 -0300 Subject: [Numpy-discussion] add NPY_UNUSED to _import_{array|umath} Message-ID: When using gcc -Wall, the warnings below are common when including NumPy headers (e.g. Cython testsuite) /usr/lib64/python2.6/site-packages/numpy/core/include/numpy/__multiarray_api.h:968: warning: ?_import_array? defined but not used /usr/lib64/python2.6/site-packages/numpy/core/include/numpy/__ufunc_api.h:182: warning: ?_import_umath? defined but not used Any chance these functions could be decorated with the NPY_UNUSED() macro ? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From mdroe at stsci.edu Wed Aug 18 15:41:55 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 18 Aug 2010 15:41:55 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AC691.4000504@stsci.edu> Message-ID: <4C6C3783.5040307@stsci.edu> On 08/17/2010 08:10 PM, Ralf Gommers wrote: > > > On Wed, Aug 18, 2010 at 3:08 AM, Charles R Harris > > wrote: > > > > On Tue, Aug 17, 2010 at 11:27 AM, Michael Droettboom > > wrote: > > I'm getting one unit test error on SunOS 5.8 UltraSPARC 5, > with Sun WorkShop 6 update 2 C 5.3 Patch 111679-14 2004/02/20: > > ====================================================================== > ERROR: test_ldexp (test_umath.TestLdexp) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/mdroe/usr/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", > line 388, in test_ldexp > assert_almost_equal(ncu.ldexp(np.array(2., np.longdouble), > np.array(3, np.int16)), 16.) > TypeError: function not supported for these types, and can't > coerce safely to supported types > > Any thoughts? > > Mike > > > Does it worki for other SunOS versions? Can you pin down a bit > more which types work and which not? I suspect that the longdouble > may be a problem, but that's just because quad precision is mostly > a Sun (Oracle?) thing. > > > It's http://projects.scipy.org/numpy/ticket/1499. Fixed for Windows in > r8625, can you check if the same fix works on SunOS? This fix does not seem to resolve the test. Mike -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Wed Aug 18 17:28:53 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 18 Aug 2010 17:28:53 -0400 Subject: [Numpy-discussion] Seeking advice on crowded namespace. In-Reply-To: References: <4C6AF47D.6070900@gmail.com> <4C6C0419.10802@gmail.com> Message-ID: On 2010-08-18, at 12:36 PM, Charles R Harris wrote: > In general it is a good idea to keep the specific bits out of classes since designing *the* universal class is hard and anyone who wants to just borrow a bit of code will end up cursing the SOB who buried the good stuff in a class, creating all sorts of inconvenient dependencies. That's my experience, anyway. Mine too. I can feel slightly less ashamed of my gut feeling in this regard knowing that it's shared by others. :) I took think the "functional" interface ought to be kept for those who really want it, even if they have to separately import it. David From zbyszek at in.waw.pl Wed Aug 18 17:31:56 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Wed, 18 Aug 2010 23:31:56 +0200 Subject: [Numpy-discussion] numpy.concatenate slower than slice copying In-Reply-To: References: <20100817130310.GA12292@in.waw.pl> Message-ID: <20100818213156.GB12292@in.waw.pl> On Wed, Aug 18, 2010 at 02:06:51AM +0100, Francesc Alted wrote: > Hey Zbyszek, > > 2010/8/17, Zbyszek Szmek : > > Hi, > > this is a problem which came up when trying to replace a hand-written > > array concatenation with a call to numpy.vstack: > > for some array sizes, > > > > numpy.vstack(data) > > > > runs > 20% longer than a loop like > > > > alldata = numpy.empty((tlen, dim)) > > for x in data: > > step = x.shape[0] > > alldata[pos:pos+step] = x > > pos += step > > > > (example script attached) > [clip] > > I was curious on what is happening here, so after some profiling with > cachegrind, I've come to the conclusion that `numpy.concatenate` is > using the `memcpy` system call so as to copy data from sources to > recipient. On his hand, your `concat` function is making use of the > `__setitem__` method of ndarray, which does not use `memcpy` (this is > probably due to the fact that it has to deal with strides). Hey Francesc, thank you for your detailed answer. It seems that memcpy which should always be faster then memmove is sometimes slower! What happens is that using the slice assignment calls memmove() which calls _wordcopy_fwd_aligned() [1] which is apparently faster than memcpy() [2] [1] http://www.eglibc.org/cgi-bin/viewcvs.cgi/trunk/libc/string/wordcopy.c?rev=77&view=auto [2] http://www.eglibc.org/cgi-bin/viewcvs.cgi/trunk/libc/sysdeps/x86_64/memcpy.S?rev=11186&view=markup I guess that you're not seeing the difference because I'm using an amd64 specific memcpy written in assembly, and you're using a i586 implementation. I've tried to reproduce the problem in a C program, but there the memcpy is always much faster than memmove, as should be. I've verified that the difference between memcpy and memmove is the problem by patching array_concatenate to always use memmove: diff --git a/numpy/core/src/multiarray/multiarraymodule.c b/numpy/core/src/multiarray/multia index de63f33..e7f8643 100644 --- a/numpy/core/src/multiarray/multiarraymodule.c +++ b/numpy/core/src/multiarray/multiarraymodule.c @@ -437,7 +437,7 @@ PyArray_Concatenate(PyObject *op, int axis) data = ret->data; for (i = 0; i < n; i++) { numbytes = PyArray_NBYTES(mps[i]); - memcpy(data, mps[i]->data, numbytes); + memmove(data, mps[i]->data, numbytes); data += numbytes; } which gives a speedup the same as using the slice assignment: zbyszek at ameba ~/mdp/tmp % python2.6 del_cum3.py numpy 10000 1000 10 10 problem size: (10000x1000) x 10 = 10^8 0.814s <----- without the patch zbyszek at ameba ~/mdp/tmp % PYTHONPATH=/var/tmp/install/lib/python2.6/site-packages python2.6 del_cum3.py numpy 10000 1000 10 10 problem size: (10000x1000) x 10 = 10^8 0.637s <----- with the stupid patch > Now, it turns out that `memcpy` may be not optimal for every platform, > and a direct fetch and assign approach could be sometimes faster. My > guess is that this is what is happening in your case. On my machine, > running latest Ubuntu Linux, I'm not seeing this difference though: > > faltet at ubuntu:~/carray$ python bench/concat.py numpy 1000 1000 10 3 > problem size: (1000x1000) x 10 = 10^7 > 0.247s > faltet at ubuntu:~/carray$ python bench/concat.py concat 1000 1000 10 3 > problem size: (1000x1000) x 10 = 10^7 > 0.246s > > and neither when running Windows (XP): > > C:\tmp>python del_cum3.py numpy 10000 1000 1 10 > problem size: (10000x1000) x 1 = 10^7 > 0.227s > > C:\tmp>python del_cum3.py concat 10000 1000 1 10 > problem size: (10000x1000) x 1 = 10^7 > 0.223s Probably the architecture (and thus glibc implementation) is more important than the operating system. But the problem is very much dependent on the size of the arrays, so probably on aligment and other details N.B. with the the parameters above I get 0.081s vs. 0.066s. > Now the new method (carray) with compression level 1 (note the new > parameter at the end of the command line): > > faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py carray > 1000000 10 3 1 > problem size: (1000000) x 10 = 10^7 > time for concat: 0.186s > size of the final container: 5.076 MB This looks very interesting! Do you think it would be possible to automatically 'guess' if such compression makes sense and just use it behind the scenes as 'decompress-on-write'? I'll try to do some benchmarking tomorrow... Thanks, Zbyszek From friedrichromstedt at gmail.com Wed Aug 18 17:34:09 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 18 Aug 2010 23:34:09 +0200 Subject: [Numpy-discussion] fancy indexing question In-Reply-To: References: Message-ID: 2010/8/17 Angus McMorland : > a = np.random.randint(10, size=(4,3,2)) > ord = np.array([[3,2,1,0],[0,2,1,3],[3,0,2,1]]).T > b = a[ord][:,np.eye(3, dtype=bool),:] b = a[ord, arange(0, 3)] Broadcasting rules! Tested. Friedrich From friedrichromstedt at gmail.com Wed Aug 18 17:42:28 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 18 Aug 2010 23:42:28 +0200 Subject: [Numpy-discussion] Problem with loadtxt In-Reply-To: References: <4C6B25C3.40204@enthought.com> Message-ID: 2010/8/18 Scott MacDonald : > In [42]: c = StringIO("5399354557888517120") Well, should't it be StringIO("5399354557888517312")? Maybe I'm missing sth. Friedrich From scott.p.macdonald at gmail.com Wed Aug 18 18:01:59 2010 From: scott.p.macdonald at gmail.com (Scott MacDonald) Date: Wed, 18 Aug 2010 15:01:59 -0700 Subject: [Numpy-discussion] Problem with loadtxt In-Reply-To: References: <4C6B25C3.40204@enthought.com> Message-ID: Yes it should, sorry I must have copied and pasted wrong. I did some more research and found this bug report: http://projects.scipy.org/numpy/ticket/1163 I guess this is still open? I am using numpy 1.4.1 On Wed, Aug 18, 2010 at 2:42 PM, Friedrich Romstedt < friedrichromstedt at gmail.com> wrote: > 2010/8/18 Scott MacDonald : > > In [42]: c = StringIO("5399354557888517120") > > Well, should't it be StringIO("5399354557888517312")? Maybe I'm missing > sth. > > Friedrich > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Wed Aug 18 18:52:45 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 18 Aug 2010 17:52:45 -0500 Subject: [Numpy-discussion] Problem with loadtxt In-Reply-To: References: <4C6B25C3.40204@enthought.com> Message-ID: <4C6C643D.5020802@enthought.com> Scott MacDonald wrote: > Yes it should, sorry I must have copied and pasted wrong. > > I did some more research and found this bug report: > > http://projects.scipy.org/numpy/ticket/1163 > > I guess this is still open? I am using numpy 1.4.1 > > According to that ticket, it is doing int(float(x)) when the dtype is a subclass of int. This allows it to handle text like "123.45" or "1e4" properly; passing either of those directly to int() will raise a ValueError. Perhaps something like this would work for int dtypes: try: value = int(text) except ValueError: value = int(float(text)) Warren > > On Wed, Aug 18, 2010 at 2:42 PM, Friedrich Romstedt > > wrote: > > 2010/8/18 Scott MacDonald >: > > In [42]: c = StringIO("5399354557888517120") > > Well, should't it be StringIO("5399354557888517312")? Maybe I'm > missing sth. > > Friedrich > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Wed Aug 18 19:59:02 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Aug 2010 07:59:02 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 9:28 AM, David Cournapeau wrote: > On Wed, Aug 18, 2010 at 12:23 AM, Ralf Gommers > wrote: > > I am pleased to announce the availability of the second beta of NumPy > 1.5.0. > > This will be the first NumPy release to include support for Python 3, as > > well as for Python 2.7. > > > > Please try this beta and report any problems on the NumPy mailing list. > > Especially with Python 3 testing will be very useful. On Linux and OS X > > building from source should be straightforward, for Windows a binary > > installer is provided. There is one important known issue on Windows > left, > > in fromfile and tofile (ticket 1583). > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > I saw that the beta was put on pypi as well. I think we should not do > that: easy_install will always the most recent version, and this means > trouble. > > OK, I will remove it again tonight. In this case there won't be much trouble though, the only reason this is a beta instead of an rc is because of one py3k problem - and 1.4.1 doesn't work with that (or 2.7) anyway. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Aug 18 20:00:48 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Aug 2010 08:00:48 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 9:32 AM, Bruce Southey wrote: > On Tue, Aug 17, 2010 at 7:07 PM, Ralf Gommers > wrote: > > > > > > On Wed, Aug 18, 2010 at 12:31 AM, Bruce Southey > wrote: > >> > >> On Tue, Aug 17, 2010 at 10:23 AM, Ralf Gommers > >> wrote: > >> > I am pleased to announce the availability of the second beta of NumPy > >> > 1.5.0. > >> > This will be the first NumPy release to include support for Python 3, > as > >> > well as for Python 2.7. > >> > > >> > Please try this beta and report any problems on the NumPy mailing > list. > >> > Especially with Python 3 testing will be very useful. On Linux and OS > X > >> > building from source should be straightforward, for Windows a binary > >> > installer is provided. There is one important known issue on Windows > >> > left, > >> > in fromfile and tofile (ticket 1583). > >> > > >> > Binaries, sources and release notes can be found at > >> > https://sourceforge.net/projects/numpy/files/ > >> > > >> > Enjoy, > >> > Ralf > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> > >> Hi, > >> How do you get nose installed under Windows with Python3.1 such as a > >> single package or command? > > > > > > There is a nose branch that works with py3k here: > > http://bitbucket.org/jpellerin/nose3/. Just install with "python > setup.py > > install". > > > > I will try not to forget to mention this in the next announcement. > > > > Cheers, > > Ralf > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > Okay, > While I did not want to use hg, it was not very painful... > > It did result in 38 errors (given below) but I did not see anything > that I would suggest a numpy error. All these seem to be Windows file > access related problems which I did not see on the same machine: > NumPy version 1.5.0b2 > NumPy is installed in E:\Python26\lib\site-packages\numpy > Python version 2.6.3 (r263rc1:75186, Oct 2 2009, 20:40:30) [MSC > v.1500 32 bit (Intel)] > nose version 0.11.1 > > So it could be nose related. > > I don't think it is: >>> np.fromfile('fromfile.txt') Traceback (most recent call last): File "", line 1, in IOError: first argument must be an open file >>> f = open('fromfile.txt') >>> np.fromfile(f) Traceback (most recent call last): File "", line 1, in IOError: first argument must be an open file >>> f.read() '0.0 1.0\n' fromfile and tofile have a real problem. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Thu Aug 19 03:44:23 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 19 Aug 2010 00:44:23 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 beta 2 In-Reply-To: References: <4C6AE51A.40202@uci.edu> <4C6B4F70.90301@uci.edu> <4C6B7B1B.5070704@uci.edu> Message-ID: <4C6CE0D7.1000401@uci.edu> On 8/18/2010 6:00 AM, Charles R Harris wrote: > > > On Wed, Aug 18, 2010 at 6:50 AM, Charles R Harris > > wrote: > > > > On Wed, Aug 18, 2010 at 12:18 AM, Christoph Gohlke > wrote: > > > > On 8/17/2010 9:56 PM, Charles R Harris wrote: > > > > > > On Tue, Aug 17, 2010 at 9:11 PM, Christoph Gohlke > > > >> wrote: > > > > > > > > On 8/17/2010 1:02 PM, Charles R Harris wrote: > > > > > > > > > On Tue, Aug 17, 2010 at 1:38 PM, Christoph Gohlke > > > > > > > > >>> wrote: > > > > > > > > > > > > On 8/17/2010 8:23 AM, Ralf Gommers wrote: > > > > > > I am pleased to announce the availability of the second > > beta of > > > NumPy > > > 1.5.0. This will be the first NumPy release to include > > support for > > > Python 3, as well as for Python 2.7. > > > > > > Please try this beta and report any problems on the > NumPy > > > mailing list. > > > Especially with Python 3 testing will be very > useful. On Linux > > > and OS X > > > building from source should be straightforward, for > > Windows a binary > > > installer is provided. There is one important known > issue > > on Windows > > > left, in fromfile and tofile (ticket 1583). > > > > > > Binaries, sources and release notes can be found at > > > https://sourceforge.net/projects/numpy/files/ > > > > > > > > > Enjoy, > > > Ralf > > > > > > > > > NumPy 1.5.0 beta 2 built with msvc9/mkl for Python 2.7 > and 3.1 (32 > > > and 64 bit) still reports many (> 200) warnings and three > > known test > > > failures/errors. Nothing serious, but it would be nice > to clean up > > > before the final release. > > > > > > The warnings are of the type "Warning: invalid value > > encountered in" > > > for the functions reduce, fmax, fmin, logaddexp, > maximum, greater, > > > less_equal, greater_equal, absolute, and others. I do > not see > > any of > > > these warnings in the msvc9 builds of numpy 1.4.1. > > > > > > > > > The warnings were accidentally turned off for earlier > versions of > > Numpy. > > > I expect these warnings are related to nans and probably due to > > problems > > > with isnan or some such. Can you take a closer look? The > fmax function > > > should be easy to check out. > > > > > > > > > > > > Chuck > > > > > > > > > Thanks for the hint. Warnings are issued in the test_umath > test_*nan* > > functions. The problem can be condensed to this statement: > > > > > >> numpy.array([numpy.nan]) > 0 > > Warning: invalid value encountered in greater > > array([False], dtype=bool) > > > > > > When using msvc, ordered comparisons involving NaN raise > an exception > > [1], i.e. set the 'invalid' x87 status bit, which leads to > the warning > > being printed. I don't know if this violates IEEE 754 or > C99 standards > > but it does not happen with the gcc builds. Maybe > > seterr(invalid='ignore') could be added to the test_*nan* > functions? > > > > [1] > http://msdn.microsoft.com/en-us/library/e7s85ffb%28v=VS.90%29.aspx > > > > > > OK, this does seem to be the standard. For instance > > > > The isless macro determines whether its first argument is less > than its > > second > > argument. The value of isless(x, y) is always equal to (x) < > (y); however, > > unlike (x) < (y), isless(x, y) does not raise the ??invalid?? > floating-point > > exception when x and y are unordered. > > > > There are other macros for the rest of the comparisons. Can > you check if > > MSVC has these macros available? > > > > MSVC doesn't have these macros. It should not be too difficult > to define > them according to > http://code.google.com/p/v8/source/browse/branches/bleeding_edge/src/platform-win32.cc > > int isless(double x, double y) { > return isnan(x) || isnan(y) ? 0 : x < y; > } > > > int isgreater(double x, double y) { > return isnan(x) || isnan(y) ? 0 : x > y; > } > > > Looks like we should just rewrite the ufuncs. Having these checks > will slow things down a bit but it is probably the easiest way to > make things portable without doing a lot of complicated work to > determine what the platform does. However, this means that numpy > won't ever set exceptions when nans are encountered and this is a > design decision we should make up front. > > > Thinking about it a bit more, I don't think we should fix the > comparisons, rather we should fix the tests to ignore the warnings. > OTOH, since we already have defined behaviors for max/min/fmax/fmin > maybe we should fix those functions. Or maybe not, since warning on > invalid values is orthogonal to what we do with them. Hmm... > > Chuck > I opened a ticket and attached a patch that silences the warnings during the tests. http://projects.scipy.org/numpy/ticket/1587 -- Christoph From gregwh at gmail.com Thu Aug 19 10:01:02 2010 From: gregwh at gmail.com (greg whittier) Date: Thu, 19 Aug 2010 10:01:02 -0400 Subject: [Numpy-discussion] summing over more than one axis Message-ID: I frequently deal with 3D data and would like to sum (or find the mean, etc.) over the last two axes. I.e. sum a[i,j,k] over j and k. I find using .sum() really convenient for 2d arrays but end up reshaping 2d arrays to do this. I know there has to be a more convenient way. Here's what I'm doing a = np.arange(27).reshape(3,3,3) # sum over axis 1 and 2 result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) Is there a cleaner way to do this? I'm sure I'm missing something obvious. Thanks, Greg From geometrian at gmail.com Thu Aug 19 10:05:16 2010 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 19 Aug 2010 08:05:16 -0600 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: Hi, Couldn't you do it with several sum steps? E.g.: result = array.sum(axis=1).sum(axis=2) Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Thu Aug 19 10:06:32 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Thu, 19 Aug 2010 09:06:32 -0500 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: > I frequently deal with 3D data and would like to sum (or find the > mean, etc.) over the last two axes. I.e. sum a[i,j,k] over j and k. > I find using .sum() really convenient for 2d arrays but end up > reshaping 2d arrays to do this. I know there has to be a more > convenient way. Here's what I'm doing > > a = np.arange(27).reshape(3,3,3) > > # sum over axis 1 and 2 > result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) > > Is there a cleaner way to do this? I'm sure I'm missing something obvious. > > Thanks, > Greg > Using two sums np.sum(np.sum(a, axis=-2), axis=1) -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From amcmorl at gmail.com Thu Aug 19 10:12:18 2010 From: amcmorl at gmail.com (Angus McMorland) Date: Thu, 19 Aug 2010 10:12:18 -0400 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: On 19 August 2010 10:01, greg whittier wrote: > I frequently deal with 3D data and would like to sum (or find the > mean, etc.) over the last two axes. ?I.e. sum a[i,j,k] over j and k. > I find using .sum() really convenient for 2d arrays but end up > reshaping 2d arrays to do this. ?I know there has to be a more > convenient way. ?Here's what I'm doing > > a = np.arange(27).reshape(3,3,3) > > # sum over axis 1 and 2 > result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) > > Is there a cleaner way to do this? ?I'm sure I'm missing something obvious. Another rank-generic approach is to use apply_over_axes (you get a different shape to the result this way): a = np.random.randint(20, size=(4,3,5)) b = np.apply_over_axes(np.sum, a, [1,2]).flat assert( np.all( b == a.sum(axis=2).sum(axis=1) ) ) > Thanks, > Greg > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From ralf.gommers at googlemail.com Thu Aug 19 10:44:08 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 Aug 2010 22:44:08 +0800 Subject: [Numpy-discussion] inconsistency in 10**(-2) or 10**array([-2]) In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 9:40 PM, Mark Bakker wrote: > I understand why numpy does it, but even Python itself gives 10**-2 = 0.01. > So I am wondering whether this is the intended behavior of numpy. I don't > really think so, but I may be wrong. > > Was intended, but should probably be changed: http://projects.scipy.org/numpy/ticket/301 Would be good to make this decision and get it into 1.5.0. Cheers, Ralf > > Roberto wrote: > > I think the problem is that array([-2]) is an array of integers, so the > result is also an array of integers. It works fine with array([-2.0]). > > On Wed, Aug 18, 2010 at 2:42 PM, Mark Bakker wrote: > >> Hello list, >> >> When I do 10**-2, Python nicely returns 0.02 >> >> But with numpy (version 1.3.0), I get zero: >> >> In [492]: 10**array([-2]) >> Out[492]: array([0]) >> >> Is this the intended behavior? >> >> Thanks, Mark >> >> >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregwh at gmail.com Thu Aug 19 10:57:19 2010 From: gregwh at gmail.com (greg whittier) Date: Thu, 19 Aug 2010 10:57:19 -0400 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 10:12 AM, Angus McMorland wrote: > Another rank-generic approach is to use apply_over_axes (you get a > different shape to the result this way): > > a = np.random.randint(20, size=(4,3,5)) > b = np.apply_over_axes(np.sum, a, [1,2]).flat > assert( np.all( b == a.sum(axis=2).sum(axis=1) ) ) > Thanks for the responses! This looks like what I've been looking for. There's really a lot to numpy. It's very powerful and I feel like I've only scratched the surface. From dave.hirschfeld at gmail.com Thu Aug 19 11:08:43 2010 From: dave.hirschfeld at gmail.com (Dave) Date: Thu, 19 Aug 2010 15:08:43 +0000 (UTC) Subject: [Numpy-discussion] summing over more than one axis References: Message-ID: greg whittier gmail.com> writes: > > a = np.arange(27).reshape(3,3,3) > > # sum over axis 1 and 2 > result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) > > Is there a cleaner way to do this? I'm sure I'm missing something obvious. > Only slightly more convenient is: a.reshape(a.shape[0], -1).sum(-1) HTH, Dave From jh at physics.ucf.edu Thu Aug 19 11:29:40 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Thu, 19 Aug 2010 11:29:40 -0400 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: On Thu, 19 Aug 2010 09:06:32 -0500, G?khan Sever wrote: >On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: > >> I frequently deal with 3D data and would like to sum (or find the >> mean, etc.) over the last two axes. I.e. sum a[i,j,k] over j and k. >> I find using .sum() really convenient for 2d arrays but end up >> reshaping 2d arrays to do this. I know there has to be a more >> convenient way. Here's what I'm doing >> >> a = np.arange(27).reshape(3,3,3) >> >> # sum over axis 1 and 2 >> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) >> >> Is there a cleaner way to do this? I'm sure I'm missing something obvious. >> >> Thanks, >> Greg >> > >Using two sums > >np.sum(np.sum(a, axis=-2), axis=1) Be careful. This works for sums, but not for operations like median; the median of the row medians may not be the global median. So, you need to do the medians in one step. I'm not aware of a method cleaner than manually reshaping first. There may also be speed reasons to do things in one step. But, two steps may look cleaner in code. --jh-- From jsalvati at u.washington.edu Thu Aug 19 13:00:00 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 10:00:00 -0700 Subject: [Numpy-discussion] loading database data into numpy arrays Message-ID: Hello, I am trying to load some time series data into numpy arrays from a MySQL database using pyodbc, but I am not sure what the standard way to do this is. I found the following: http://www.aidanfindlater.com/python-db-api-to-numpy but when I tried after doing a select (I made sure the cursor had rows). np.fromiter(cursor, dtype=[('a',float), ('b', float)]) I got Traceback (most recent call last): File "", line 1, in File "C:\Users\jsalvatier\workspace\rpxdata\src\rpxdata\mysql.py", line 45, in run_statistics_15 return np.fromiter(cursor, dtype=[('a',float), ('b', float)]) TypeError: expected a readable buffer object Does anyone have advice on this? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Thu Aug 19 13:18:29 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 19 Aug 2010 10:18:29 -0700 Subject: [Numpy-discussion] loading database data into numpy arrays In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 10:00 AM, John Salvatier wrote: > Hello, > > I am trying to load some time series data into numpy arrays from a MySQL > database using pyodbc, but I am not sure what the standard way to do this > is. I found the following: > http://www.aidanfindlater.com/python-db-api-to-numpy but when I tried after > doing a select (I made sure the cursor had rows). > > np.fromiter(cursor, dtype=[('a',float), ('b', float)]) > > I got > > Traceback (most recent call last): > ? File "", line 1, in > ? File "C:\Users\jsalvatier\workspace\rpxdata\src\rpxdata\mysql.py", line > 45, in run_statistics_15 > ??? return np.fromiter(cursor, dtype=[('a',float), ('b', float)]) > TypeError: expected a readable buffer object > > Does anyone have advice on this? How about np.asarray(cursor.fetchall())? From jsalvati at u.washington.edu Thu Aug 19 13:23:22 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 10:23:22 -0700 Subject: [Numpy-discussion] loading database data into numpy arrays In-Reply-To: References: Message-ID: That worked! Thanks! On Thu, Aug 19, 2010 at 10:18 AM, Keith Goodman wrote: > On Thu, Aug 19, 2010 at 10:00 AM, John Salvatier > wrote: > > Hello, > > > > I am trying to load some time series data into numpy arrays from a MySQL > > database using pyodbc, but I am not sure what the standard way to do this > > is. I found the following: > > http://www.aidanfindlater.com/python-db-api-to-numpy but when I tried > after > > doing a select (I made sure the cursor had rows). > > > > np.fromiter(cursor, dtype=[('a',float), ('b', float)]) > > > > I got > > > > Traceback (most recent call last): > > File "", line 1, in > > File "C:\Users\jsalvatier\workspace\rpxdata\src\rpxdata\mysql.py", line > > 45, in run_statistics_15 > > return np.fromiter(cursor, dtype=[('a',float), ('b', float)]) > > TypeError: expected a readable buffer object > > > > Does anyone have advice on this? > > How about np.asarray(cursor.fetchall())? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Thu Aug 19 13:37:01 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 19 Aug 2010 10:37:01 -0700 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 47, Issue 47 In-Reply-To: References: Message-ID: Date: Wed, 18 Aug 2010 09:20:41 +0200 > From: Sturla Molden > Subject: Re: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John > Hunter) > To: Discussion of Numerical Python > Message-ID: <8C0B2317-2A22-4828-99E8-AC6C0F77824F at molden.no> > Content-Type: text/plain; charset=us-ascii; format=flowed; > delsp=yes > > Den 18. aug. 2010 kl. 08.19 skrev Martin Raspaud > : > > > Once upon a time, when my boss wanted me to use matlab, I found myself > > implementing a python interpreter in matlab... > > > > There are just two sane solutions for Matlab: Either embed CPyton in a > MEX file, or use Matlab's JVM to run Jython ;) > > http://vader.cse.lehigh.edu/~perkins/pymex.html > > Sturla > > Date: Wed, 18 Aug 2010 09:33:59 +0100 > From: Robin > Subject: Re: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John > Hunter) > To: Discussion of Numerical Python > Message-ID: > > > > Content-Type: text/plain; charset=ISO-8859-1 > Just thought I'd mention another one since this came up: > > http://github.com/kw/pymex > This one works very nicely - it proxies any Python objects so you can > use, should you want to, the Matlab IDE as a python interpreter, > supports numpy arrays etc. Also cross-platform - I even got it to work > with 64 bit matlab/python on windows (in a fork on github). > > Thanks for the ideas; are any/all of these "solutions" freely/easily redistributable? DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Aug 19 15:13:16 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 19 Aug 2010 15:13:16 -0400 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 11:29 AM, Joe Harrington wrote: > On Thu, 19 Aug 2010 09:06:32 -0500, G?khan Sever wrote: > >>On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: >> >>> I frequently deal with 3D data and would like to sum (or find the >>> mean, etc.) over the last two axes. ?I.e. sum a[i,j,k] over j and k. >>> I find using .sum() really convenient for 2d arrays but end up >>> reshaping 2d arrays to do this. ?I know there has to be a more >>> convenient way. ?Here's what I'm doing >>> >>> a = np.arange(27).reshape(3,3,3) >>> >>> # sum over axis 1 and 2 >>> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) >>> >>> Is there a cleaner way to do this? ?I'm sure I'm missing something obvious. >>> >>> Thanks, >>> Greg >>> >> >>Using two sums >> >>np.sum(np.sum(a, axis=-2), axis=1) > > Be careful. ?This works for sums, but not for operations like median; > the median of the row medians may not be the global median. ?So, you > need to do the medians in one step. ?I'm not aware of a method cleaner > than manually reshaping first. ?There may also be speed reasons to do > things in one step. ?But, two steps may look cleaner in code. I think, two .sums() are the most accurate, if precision matters. One big summation is often not very precise. Josef > > --jh-- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jsalvati at u.washington.edu Thu Aug 19 16:03:29 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 13:03:29 -0700 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: Precise in what sense? Numerical accuracy? If so, why is that? On Thu, Aug 19, 2010 at 12:13 PM, wrote: > On Thu, Aug 19, 2010 at 11:29 AM, Joe Harrington > wrote: > > On Thu, 19 Aug 2010 09:06:32 -0500, G?khan Sever > wrote: > > > >>On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: > >> > >>> I frequently deal with 3D data and would like to sum (or find the > >>> mean, etc.) over the last two axes. I.e. sum a[i,j,k] over j and k. > >>> I find using .sum() really convenient for 2d arrays but end up > >>> reshaping 2d arrays to do this. I know there has to be a more > >>> convenient way. Here's what I'm doing > >>> > >>> a = np.arange(27).reshape(3,3,3) > >>> > >>> # sum over axis 1 and 2 > >>> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) > >>> > >>> Is there a cleaner way to do this? I'm sure I'm missing something > obvious. > >>> > >>> Thanks, > >>> Greg > >>> > >> > >>Using two sums > >> > >>np.sum(np.sum(a, axis=-2), axis=1) > > > > Be careful. This works for sums, but not for operations like median; > > the median of the row medians may not be the global median. So, you > > need to do the medians in one step. I'm not aware of a method cleaner > > than manually reshaping first. There may also be speed reasons to do > > things in one step. But, two steps may look cleaner in code. > > I think, two .sums() are the most accurate, if precision matters. One > big summation is often not very precise. > > Josef > > > > > > --jh-- > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Thu Aug 19 16:58:09 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 13:58:09 -0700 Subject: [Numpy-discussion] loading database data into numpy arrays In-Reply-To: References: Message-ID: I found the cause of the original problem. pyodbc.cursor.next() returns a pyodbc.Row object, which numpy does not seem to know how to deal with but is essentially an extended tuple. Therefore in order to use the fromiter method to make a numpy structured array from a mysql or other db query, you must change the iterator slightly. I did the following: The generator expression simply casts the Row object to a tuple. cursor = conn.cursor() cursor.execute('SELECT a, b FROM demo') result = np.fromiter((tuple (row) for row in cursor), dtype=[('a',float), ('b', float)]) On Thu, Aug 19, 2010 at 10:23 AM, John Salvatier wrote: > That worked! Thanks! > > > On Thu, Aug 19, 2010 at 10:18 AM, Keith Goodman wrote: > >> On Thu, Aug 19, 2010 at 10:00 AM, John Salvatier >> wrote: >> > Hello, >> > >> > I am trying to load some time series data into numpy arrays from a MySQL >> > database using pyodbc, but I am not sure what the standard way to do >> this >> > is. I found the following: >> > http://www.aidanfindlater.com/python-db-api-to-numpy but when I tried >> after >> > doing a select (I made sure the cursor had rows). >> > >> > np.fromiter(cursor, dtype=[('a',float), ('b', float)]) >> > >> > I got >> > >> > Traceback (most recent call last): >> > File "", line 1, in >> > File "C:\Users\jsalvatier\workspace\rpxdata\src\rpxdata\mysql.py", >> line >> > 45, in run_statistics_15 >> > return np.fromiter(cursor, dtype=[('a',float), ('b', float)]) >> > TypeError: expected a readable buffer object >> > >> > Does anyone have advice on this? >> >> How about np.asarray(cursor.fetchall())? >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From emma.willemsma at morgansolar.com Thu Aug 19 17:07:03 2010 From: emma.willemsma at morgansolar.com (Emma Willemsma) Date: Thu, 19 Aug 2010 17:07:03 -0400 Subject: [Numpy-discussion] Using numpy's flatten_dtype with structured dtypes that have titles In-Reply-To: References: Message-ID: Hi, I originally posted this question on stackoverflow, and the user who responded to me suggested that I bring this issue to the attention of the numpy developers. I am working with structured arrays to store experimental data. I'm using titles to store information about my fields, in this case the units of measure. When I call numpy.lib.io.flatten_dtype() on my dtype, I get: ValueError: too many values to unpack File "c:\Python25\Lib\site-packages\numpy\lib\_iotools.py", line 78, in flatten_dtype (typ, _) = ndtype.fields[field] I wouldn't really care, except that numpy.genfromtxt() calls numpy.lib.io.flatten_dtype(), and I need to be able to import my data from text files. I'm wondering if I've done something wrong. Is flatten_dtype() not meant to support titles? Is there a work-around for genfromtxt()? Here's a snippet of my code: import numpy fname = "C:\\Somefile.txt" dtype = numpy.dtype([(("Amps","Current"),"f8"),(("Volts","Voltage"),"f8")]) myarray = numpy.genfromtxt(fname,dtype) The user who responded suggested the following monkey-patch: import numpy import numpy.lib.io def flatten_dtype(ndtype): """ Unpack a structured data-type. """ names = ndtype.names if names is None: return [ndtype] else: types = [] for field in names: typ_fields = ndtype.fields[field] flat_dt = flatten_dtype(typ_fields[0]) types.extend(flat_dt) return types numpy.lib.io.flatten_dtype=flatten_dtype Is this something that could be changed in numpy? Thanks! Emma -------------- next part -------------- An HTML attachment was scrubbed... URL: From jerlark386 at gmail.com Thu Aug 19 17:08:57 2010 From: jerlark386 at gmail.com (Jeremiah) Date: Thu, 19 Aug 2010 17:08:57 -0400 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: Word size. imagine 2.123456789 vs 22.12345679 the more that can stored to the right of the decimal, the more precise. That larger the number is on the left, the less that can stored on the right. On Thu, Aug 19, 2010 at 4:03 PM, John Salvatier wrote: > Precise in what sense? Numerical accuracy? If so, why is that? > > On Thu, Aug 19, 2010 at 12:13 PM, wrote: >> >> On Thu, Aug 19, 2010 at 11:29 AM, Joe Harrington >> wrote: >> > On Thu, 19 Aug 2010 09:06:32 -0500, G?khan Sever >> > wrote: >> > >> >>On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: >> >> >> >>> I frequently deal with 3D data and would like to sum (or find the >> >>> mean, etc.) over the last two axes. ?I.e. sum a[i,j,k] over j and k. >> >>> I find using .sum() really convenient for 2d arrays but end up >> >>> reshaping 2d arrays to do this. ?I know there has to be a more >> >>> convenient way. ?Here's what I'm doing >> >>> >> >>> a = np.arange(27).reshape(3,3,3) >> >>> >> >>> # sum over axis 1 and 2 >> >>> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) >> >>> >> >>> Is there a cleaner way to do this? ?I'm sure I'm missing something >> >>> obvious. >> >>> >> >>> Thanks, >> >>> Greg >> >>> >> >> >> >>Using two sums >> >> >> >>np.sum(np.sum(a, axis=-2), axis=1) >> > >> > Be careful. ?This works for sums, but not for operations like median; >> > the median of the row medians may not be the global median. ?So, you >> > need to do the medians in one step. ?I'm not aware of a method cleaner >> > than manually reshaping first. ?There may also be speed reasons to do >> > things in one step. ?But, two steps may look cleaner in code. >> >> I think, two .sums() are the most accurate, if precision matters. One >> big summation is often not very precise. >> >> Josef >> >> >> > >> > --jh-- >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Thu Aug 19 17:20:03 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 19 Aug 2010 17:20:03 -0400 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 4:03 PM, John Salvatier wrote: > Precise in what sense? Numerical accuracy? If so, why is that? I don't remember where I ran into this example, maybe integer underflow (?) with addition. NIST ANOVA test cases have some nasty badly scaled variables but I have problems creating one, difference in 10th or higher digit >>> a = 1000000*np.random.randn(10000,1000) >>> a.sum() -820034796.05545747 >>> np.sort(a.ravel())[::-1].sum() -820034795.87886333 >>> np.sort(a.ravel()).sum() -820034795.88172638 >>> np.sort(a,0)[::-1].sum() -820034795.82333243 >>> np.sort(a,1)[::-1].sum() -820034796.05559027 >>> a.sum(-1).sum(-1) -820034796.05551744 >>> np.sort(a,1)[::-1].sum(-1).sum(-1) -820034796.05543578 >>> np.sort(a,0)[::-1].sum(-1).sum(-1) -820034796.05590343 >>> np.sort(a,1).sum(-1).sum(-1) -820034796.05544424 >>> am = a.mean() >>> am*a.size + np.sort(a-am,1).sum(-1).sum(-1) -820034796.05554879 >>> a.size * np.sort(a,1).mean(-1).mean(-1) -820034796.05544722 badly scaled or badly sorted arrays don't add up well but I'm not able to get worse than 10th or 11th decimal in some random generated examples with size 10000x1000 Josef > > On Thu, Aug 19, 2010 at 12:13 PM, wrote: >> >> On Thu, Aug 19, 2010 at 11:29 AM, Joe Harrington >> wrote: >> > On Thu, 19 Aug 2010 09:06:32 -0500, G?khan Sever >> > wrote: >> > >> >>On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: >> >> >> >>> I frequently deal with 3D data and would like to sum (or find the >> >>> mean, etc.) over the last two axes. ?I.e. sum a[i,j,k] over j and k. >> >>> I find using .sum() really convenient for 2d arrays but end up >> >>> reshaping 2d arrays to do this. ?I know there has to be a more >> >>> convenient way. ?Here's what I'm doing >> >>> >> >>> a = np.arange(27).reshape(3,3,3) >> >>> >> >>> # sum over axis 1 and 2 >> >>> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) >> >>> >> >>> Is there a cleaner way to do this? ?I'm sure I'm missing something >> >>> obvious. >> >>> >> >>> Thanks, >> >>> Greg >> >>> >> >> >> >>Using two sums >> >> >> >>np.sum(np.sum(a, axis=-2), axis=1) >> > >> > Be careful. ?This works for sums, but not for operations like median; >> > the median of the row medians may not be the global median. ?So, you >> > need to do the medians in one step. ?I'm not aware of a method cleaner >> > than manually reshaping first. ?There may also be speed reasons to do >> > things in one step. ?But, two steps may look cleaner in code. >> >> I think, two .sums() are the most accurate, if precision matters. One >> big summation is often not very precise. >> >> Josef >> >> >> > >> > --jh-- >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From thusweare at hotmail.com Thu Aug 19 17:22:32 2010 From: thusweare at hotmail.com (sm lkd) Date: Thu, 19 Aug 2010 21:22:32 +0000 Subject: [Numpy-discussion] vectorization of vectorization Message-ID: Hello, Here's my problem: for each value t of an array (from 0 to 1e6) a smaller array is computed (size between 2-6). To compute the smaller array, I have a function (which can be easily vectorized if necessary) which takes t and an array of powers of t. The return is an array of modified Bessel function values, i.e.: def funct(order, t, power): return special.iv(order, t)**power Note that order and power are arrays after this vectorization: vec_func = sp.vectorization(func) Right this is how it's used: for i in range(1000000): y[i] = vec_func(orders, t, powers).prod() Incredibly slow. Of course, it is desirable to vectorize it it terms of t. I have tried different methods but still cannot make it work. Any suggestions or ointers? Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sransom at nrao.edu Thu Aug 19 17:31:20 2010 From: sransom at nrao.edu (Scott Ransom) Date: Thu, 19 Aug 2010 17:31:20 -0400 Subject: [Numpy-discussion] vectorization of vectorization In-Reply-To: References: Message-ID: <201008191731.21053.sransom@nrao.edu> If you use already vectorized functions (like special.iv) you often don't need to use vectorization() For example: ----------------------------- import numpy as num import scipy.special as special def funct(order, t, power): return special.iv(order, t)**power order = num.arange(4.0) ts = num.linspace(0.2, 0.4, 3) funct(order, ts[:,num.newaxis], 2.0) ----------------------------- That gives: In [39]: funct(order, ts[:,num.newaxis], 2.0) Out[39]: array([[ 1.02015056e+00, 1.01004176e-02, 2.51671536e-05, 2.79169796e-08], [ 1.04576573e+00, 2.30110211e-02, 1.28473444e-04, 3.19983896e-07], [ 1.08243587e+00, 4.16269171e-02, 4.10791968e-04, 1.81365508e-06]]) With no loop. And both order and t are vectors.... Scott On Thursday, August 19, 2010 05:22:32 pm sm lkd wrote: > Hello, > > Here's my problem: for each value t of an array (from 0 to 1e6) a > smaller array is computed (size between 2-6). To compute the smaller > array, I have a function (which can be easily vectorized if necessary) > which takes t and an array of powers of t. The return is an array of > modified Bessel function values, i.e.: > > def funct(order, t, power): > return special.iv(order, t)**power > > Note that order and power are arrays after this vectorization: > vec_func = sp.vectorization(func) > > Right this is how it's used: > for i in range(1000000): > y[i] = vec_func(orders, t, powers).prod() > > Incredibly slow. > > Of course, it is desirable to vectorize it it terms of t. I have tried > different methods but still cannot make it work. Any suggestions or > ointers? > > Thank you. -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From jsalvati at u.washington.edu Thu Aug 19 17:32:42 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 14:32:42 -0700 Subject: [Numpy-discussion] vectorization of vectorization In-Reply-To: References: Message-ID: This seems simple, so perhaps I am missing something, but what about this: special.iv(orders[:, newaxis], arange(1e6)[newaxis, : ], powers[:, newaxis]).prod(axis = 2) This will probably use up a bit slow/memoryintensive, so you probably want to use numexpr to speed it up a bit. On Thu, Aug 19, 2010 at 2:22 PM, sm lkd wrote: > Hello, > > Here's my problem: for each value t of an array (from 0 to 1e6) a smaller > array is computed (size between 2-6). To compute the smaller array, I have a > function (which can be easily vectorized if necessary) which takes t and an > array of powers of t. The return is an array of modified Bessel function > values, i.e.: > > def funct(order, t, power): > return special.iv(order, t)**power > > Note that order and power are arrays after this vectorization: > vec_func = sp.vectorization(func) > > Right this is how it's used: > for i in range(1000000): > y[i] = vec_func(orders, t, powers).prod() > > Incredibly slow. > > Of course, it is desirable to vectorize it it terms of t. I have tried > different methods but still cannot make it work. Any suggestions or ointers? > > Thank you. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thusweare at hotmail.com Thu Aug 19 17:41:52 2010 From: thusweare at hotmail.com (sm lkd) Date: Thu, 19 Aug 2010 21:41:52 +0000 Subject: [Numpy-discussion] max value of an array of lists Message-ID: Hello again, Here's a simple one. I have an array/matrix of lists (floats). array.max() or max(array) returns list (!) and not an individual value. Of course, it is possible to loop, but maybe someone knows a better way to do the same. Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Thu Aug 19 17:44:14 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 14:44:14 -0700 Subject: [Numpy-discussion] max value of an array of lists In-Reply-To: References: Message-ID: You have an object array of lists? Why? On Thu, Aug 19, 2010 at 2:41 PM, sm lkd wrote: > Hello again, > > Here's a simple one. I have an array/matrix of lists (floats). array.max() > or max(array) returns list (!) and not an individual value. Of course, it is > possible to loop, but maybe someone knows a better way to do the same. > > Thanks, > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Thu Aug 19 17:45:35 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 19 Aug 2010 14:45:35 -0700 Subject: [Numpy-discussion] max value of an array of lists In-Reply-To: References: Message-ID: It sounds like you should just have a higher dimensional array. A 2 or 3 dimensional array instead of an object array. On Thu, Aug 19, 2010 at 2:44 PM, John Salvatier wrote: > You have an object array of lists? Why? > > On Thu, Aug 19, 2010 at 2:41 PM, sm lkd wrote: > >> Hello again, >> >> Here's a simple one. I have an array/matrix of lists (floats). array.max() >> or max(array) returns list (!) and not an individual value. Of course, it is >> possible to loop, but maybe someone knows a better way to do the same. >> >> Thanks, >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Thu Aug 19 17:53:00 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 19 Aug 2010 16:53:00 -0500 Subject: [Numpy-discussion] summing over more than one axis In-Reply-To: References: Message-ID: <4C6DA7BC.8050805@gmail.com> On 08/19/2010 04:20 PM, josef.pktd at gmail.com wrote: > On Thu, Aug 19, 2010 at 4:03 PM, John Salvatier > wrote: >> Precise in what sense? Numerical accuracy? If so, why is that? > I don't remember where I ran into this example, maybe integer > underflow (?) with addition. > NIST ANOVA test cases have some nasty badly scaled variables > > but I have problems creating one, difference in 10th or higher digit > >>>> a = 1000000*np.random.randn(10000,1000) >>>> a.sum() > -820034796.05545747 >>>> np.sort(a.ravel())[::-1].sum() > -820034795.87886333 >>>> np.sort(a.ravel()).sum() > -820034795.88172638 >>>> np.sort(a,0)[::-1].sum() > -820034795.82333243 >>>> np.sort(a,1)[::-1].sum() > -820034796.05559027 >>>> a.sum(-1).sum(-1) > -820034796.05551744 >>>> np.sort(a,1)[::-1].sum(-1).sum(-1) > -820034796.05543578 >>>> np.sort(a,0)[::-1].sum(-1).sum(-1) > -820034796.05590343 >>>> np.sort(a,1).sum(-1).sum(-1) > -820034796.05544424 >>>> am = a.mean() >>>> am*a.size + np.sort(a-am,1).sum(-1).sum(-1) > -820034796.05554879 >>>> a.size * np.sort(a,1).mean(-1).mean(-1) > -820034796.05544722 > > badly scaled or badly sorted arrays don't add up well > > but I'm not able to get worse than 10th or 11th decimal in some random > generated examples with size 10000x1000 > > Josef > > > >> On Thu, Aug 19, 2010 at 12:13 PM, wrote: >>> On Thu, Aug 19, 2010 at 11:29 AM, Joe Harrington >>> wrote: >>>> On Thu, 19 Aug 2010 09:06:32 -0500, G?khan Sever >>>> wrote: >>>> >>>>> On Thu, Aug 19, 2010 at 9:01 AM, greg whittier wrote: >>>>> >>>>>> I frequently deal with 3D data and would like to sum (or find the >>>>>> mean, etc.) over the last two axes. I.e. sum a[i,j,k] over j and k. >>>>>> I find using .sum() really convenient for 2d arrays but end up >>>>>> reshaping 2d arrays to do this. I know there has to be a more >>>>>> convenient way. Here's what I'm doing >>>>>> >>>>>> a = np.arange(27).reshape(3,3,3) >>>>>> >>>>>> # sum over axis 1 and 2 >>>>>> result = a.reshape((a.shape[0], a.shape[1]*a.shape[2])).sum(axis=1) >>>>>> >>>>>> Is there a cleaner way to do this? I'm sure I'm missing something >>>>>> obvious. >>>>>> >>>>>> Thanks, >>>>>> Greg >>>>>> >>>>> Using two sums >>>>> >>>>> np.sum(np.sum(a, axis=-2), axis=1) >>>> Be careful. This works for sums, but not for operations like median; >>>> the median of the row medians may not be the global median. So, you >>>> need to do the medians in one step. I'm not aware of a method cleaner >>>> than manually reshaping first. There may also be speed reasons to do >>>> things in one step. But, two steps may look cleaner in code. >>> I think, two .sums() are the most accurate, if precision matters. One >>> big summation is often not very precise. >>> >>> Josef >>> >>> You can use dtype option in many functions like sum that allow a dtype with a higher precision to be used than the input dtype. It also helps with overflow as well such as summing integers as you don't have to convert the input dtype first. However, the value very much depends on your operating system notably windows platforms that don't support highest dtypes (so float128 is not going to help over float64). Alternative use another approach to avoid loss of precision such as Python's math.fsum() http://docs.python.org/library/math.html Or Recipe 393090: Binary floating point summation accurate to full precision: http://code.activestate.com/recipes/393090/ Or Recipe 298339: More accurate sum (Python) http://code.activestate.com/recipes/298339/ These are probably more accurate than first sorting the data from low to high and then summing from low to high. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From thusweare at hotmail.com Thu Aug 19 18:36:08 2010 From: thusweare at hotmail.com (sm lkd) Date: Thu, 19 Aug 2010 22:36:08 +0000 Subject: [Numpy-discussion] vectorization of vectorization In-Reply-To: References: , Message-ID: With slight modification I got it work (fucnt as before): funct(orders, num.arange(1e6)[:, num.newaxis], powers).prod(axis = 1) This gives me exactly what I need! Thanks a lot. ps: I don't know how to post my thank you on the discussion list. Date: Thu, 19 Aug 2010 14:32:42 -0700 From: jsalvati at u.washington.edu To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] vectorization of vectorization This seems simple, so perhaps I am missing something, but what about this: special.iv(orders[:, newaxis], arange(1e6)[newaxis, : ], powers[:, newaxis]).prod(axis = 2) This will probably use up a bit slow/memoryintensive, so you probably want to use numexpr to speed it up a bit. On Thu, Aug 19, 2010 at 2:22 PM, sm lkd wrote: Hello, Here's my problem: for each value t of an array (from 0 to 1e6) a smaller array is computed (size between 2-6). To compute the smaller array, I have a function (which can be easily vectorized if necessary) which takes t and an array of powers of t. The return is an array of modified Bessel function values, i.e.: def funct(order, t, power): return special.iv(order, t)**power Note that order and power are arrays after this vectorization: vec_func = sp.vectorization(func) Right this is how it's used: for i in range(1000000): y[i] = vec_func(orders, t, powers).prod() Incredibly slow. Of course, it is desirable to vectorize it it terms of t. I have tried different methods but still cannot make it work. Any suggestions or ointers? Thank you. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From thusweare at hotmail.com Thu Aug 19 18:37:11 2010 From: thusweare at hotmail.com (sm lkd) Date: Thu, 19 Aug 2010 22:37:11 +0000 Subject: [Numpy-discussion] max value of an array of lists In-Reply-To: References: , Message-ID: No need any more. Thank you anyway! --sv Date: Thu, 19 Aug 2010 14:44:14 -0700 From: jsalvati at u.washington.edu To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] max value of an array of lists You have an object array of lists? Why? On Thu, Aug 19, 2010 at 2:41 PM, sm lkd wrote: Hello again, Here's a simple one. I have an array/matrix of lists (floats). array.max() or max(array) returns list (!) and not an individual value. Of course, it is possible to loop, but maybe someone knows a better way to do the same. Thanks, _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From eadrogue at gmx.net Thu Aug 19 18:40:03 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Fri, 20 Aug 2010 00:40:03 +0200 Subject: [Numpy-discussion] lexsort() of datetime objects doesn't work Message-ID: <20100819224003.GA8722@doriath.local> Hi, I was trying to use lexsort with an array of datetime.date objects, but it doesn't seem to work. In [86]: date = np.array([datetime.date(2000, 9, 17), datetime.date(2000, 10, 1), datetime.date(2000, 10, 22), datetime.date(2000, 11, 1)]) In [90]: date Out[90]: array([2000-09-17, 2000-10-01, 2000-10-22, 2000-11-01], dtype=object) In [91]: np.lexsort((date,)) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/ernest/src/footy/dataset.py in () ----> 1 2 3 4 5 TypeError: merge sort not available for item 0 The normal argsort works, however this is in a method that uses lexsort and I can't change it without breaking things. Ernest From charlesr.harris at gmail.com Thu Aug 19 20:03:29 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 19 Aug 2010 18:03:29 -0600 Subject: [Numpy-discussion] lexsort() of datetime objects doesn't work In-Reply-To: <20100819224003.GA8722@doriath.local> References: <20100819224003.GA8722@doriath.local> Message-ID: 2010/8/19 Ernest Adrogu? > Hi, > > I was trying to use lexsort with an array of > datetime.date objects, but it doesn't seem to work. > > In [86]: date = np.array([datetime.date(2000, 9, 17), > datetime.date(2000, 10, 1), > datetime.date(2000, 10, 22), > datetime.date(2000, 11, 1)]) > > In [90]: date > Out[90]: array([2000-09-17, 2000-10-01, 2000-10-22, 2000-11-01], > dtype=object) > > In [91]: np.lexsort((date,)) > --------------------------------------------------------------------------- > TypeError Traceback (most recent call last) > > /home/ernest/src/footy/dataset.py in () > ----> 1 > 2 > 3 > 4 > 5 > > TypeError: merge sort not available for item 0 > > The normal argsort works, however this is in a method that uses > lexsort and I can't change it without breaking things. > > The normal argsort calls the clib quicksort with a comparison function whereas the mergesort needs to be implemented for the datetime data type. It shouldn't be difficult to do, but it isn't done. I think datetime is basically int64, so it should be possible to hook things up to call that sorting routine. Please open a ticket for this, it should be fixed. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Thu Aug 19 20:19:19 2010 From: faltet at pytables.org (Francesc Alted) Date: Fri, 20 Aug 2010 01:19:19 +0100 Subject: [Numpy-discussion] index of the first element fulfilling a condition? In-Reply-To: <4C6C2174.7060905@web.de> References: <4C6C2174.7060905@web.de> Message-ID: 2010/8/18, Johann Bauer : > Hi, > > is there a good method to find the index of the first element in a 1D > array fulfilling a condition? > > The following does the job > >>>> import numpy >>>> a = numpy.array([1,5,78,3,12,4]) >>>> numpy.where( a>10 )[0][0] > 2 > > but it first compares the entire array and then selects the first index. > Is there a way (especially for big arrays) to stop the comparison after > the first hit? I'm not aware of a method in pure NumPy, but PyTables offers a solution. Consider this: In [1]: import numpy as np In [2]: import tables as tb In [3]: a = np.arange(1e7) # a sample 1-D array In [4]: ra = a.view(np.dtype([("col1", "f8")])) In [5]: ra # this is a view of the 1-D array above Out[5]: array([(0.0,), (1.0,), (2.0,), ..., (9999997.0,), (9999998.0,), (9999999.0,)], dtype=[('col1', ' 3'): ...: print row[:], row.nrow ...: break # breaks iterator when the first element fulfills the condition ...: (4.0,) 4 # element and index of the first element Hope that helps, -- Francesc Alted From faltet at pytables.org Thu Aug 19 20:35:37 2010 From: faltet at pytables.org (Francesc Alted) Date: Fri, 20 Aug 2010 01:35:37 +0100 Subject: [Numpy-discussion] numpy.concatenate slower than slice copying In-Reply-To: <20100818213156.GB12292@in.waw.pl> References: <20100817130310.GA12292@in.waw.pl> <20100818213156.GB12292@in.waw.pl> Message-ID: 2010/8/18, Zbyszek Szmek : > thank you for your detailed answer. It seems that memcpy which should always > be faster then memmove is sometimes slower! What happens is that > using the slice assignment calls memmove() which calls > _wordcopy_fwd_aligned() [1] > which is apparently faster than memcpy() [2] > > [1] > http://www.eglibc.org/cgi-bin/viewcvs.cgi/trunk/libc/string/wordcopy.c?rev=77&view=auto > [2] > http://www.eglibc.org/cgi-bin/viewcvs.cgi/trunk/libc/sysdeps/x86_64/memcpy.S?rev=11186&view=markup > > I guess that you're not seeing the difference because I'm using an > amd64 specific memcpy written in assembly, and you're using a i586 > implementation. I've tried to reproduce the problem in a C program, > but there the memcpy is always much faster than memmove, as should be. > > I've verified that the difference between memcpy and memmove is the > problem by patching array_concatenate to always use memmove: > diff --git a/numpy/core/src/multiarray/multiarraymodule.c > b/numpy/core/src/multiarray/multia > index de63f33..e7f8643 100644 > --- a/numpy/core/src/multiarray/multiarraymodule.c > +++ b/numpy/core/src/multiarray/multiarraymodule.c > @@ -437,7 +437,7 @@ PyArray_Concatenate(PyObject *op, int axis) > data = ret->data; > for (i = 0; i < n; i++) { > numbytes = PyArray_NBYTES(mps[i]); > - memcpy(data, mps[i]->data, numbytes); > + memmove(data, mps[i]->data, numbytes); > data += numbytes; > } > > which gives a speedup the same as using the slice assignment: > zbyszek at ameba ~/mdp/tmp % python2.6 del_cum3.py numpy 10000 1000 10 10 > problem size: (10000x1000) x 10 = 10^8 > 0.814s <----- without the patch > > zbyszek at ameba ~/mdp/tmp % > PYTHONPATH=/var/tmp/install/lib/python2.6/site-packages python2.6 > del_cum3.py numpy 10000 1000 10 10 > problem size: (10000x1000) x 10 = 10^8 > 0.637s <----- with the stupid patch Ok. So it is pretty clear that the flaw is a bad performance of memcpy on your platform. If you can confirm this, then would be nice if you can report that to the memcpy mantainer for the glibc project. > Probably the architecture (and thus glibc implementation) is more > important than the operating system. But the problem is very much > dependent on the size of the arrays, so probably on aligment and other > details Yes. But if memmove is faster than memcpy, then I'd say that something is wrong with memcpy. Another possibility is that the malloc in `numpy.concatenate` is different than the malloc in `numpy.empty`, and that they return memory blocks with different alignments; that could explain the difference in performance too (although this possibility is remote, IMO). >> Now the new method (carray) with compression level 1 (note the new >> parameter at the end of the command line): >> >> faltet at ubuntu:~/carray$ PYTHONPATH=. python bench/concat.py carray >> 1000000 10 3 1 >> problem size: (1000000) x 10 = 10^7 >> time for concat: 0.186s >> size of the final container: 5.076 MB > > This looks very interesting! Do you think it would be possible to > automatically 'guess' if such compression makes sense and just use > it behind the scenes as 'decompress-on-write'? I'll try to do some > benchmarking tomorrow... I'd say that, on relatively new processors (i.e. processors with around 3 MB of cache and a couple of cores or more), carray would be in general faster than a pure ndarray approach for most of cases. But indeed, benchmarking is the best way to tell. Cheers, -- Francesc Alted From erin.sheldon at gmail.com Thu Aug 19 21:05:50 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Thu, 19 Aug 2010 21:05:50 -0400 Subject: [Numpy-discussion] index of the first element fulfilling a condition? In-Reply-To: <4C6C2174.7060905@web.de> References: <4C6C2174.7060905@web.de> Message-ID: <1282266158-sup-2912@theshire> Excerpts from Johann Bauer's message of Wed Aug 18 14:07:48 -0400 2010: > Hi, > > is there a good method to find the index of the first element in a 1D > array fulfilling a condition? > > The following does the job > > >>> import numpy > >>> a = numpy.array([1,5,78,3,12,4]) > >>> numpy.where( a>10 )[0][0] > 2 > > but it first compares the entire array and then selects the first index. > Is there a way (especially for big arrays) to stop the comparison after > the first hit? > > Thanks for any help, Johann If you are willing to use scipy.weave, it is straightforward: import numpy from scipy import weave def get_first_greater(arr, value): """ Return index of first element in array that is greater than value. """ code = """ int64_t i=0, index=-1; while (i < arr.size()) { if ( arr(i) > value ) { index = i; break; } i++; } return_val = index; """ index = weave.inline(code, ['arr','value'], type_converters = weave.converters.blitz) return index Caveats: In scipy 0.7 and gcc >= 4.3 this probably won't compile. It does work for scipy 0.8. Erin Scott Sheldon From ralf.gommers at googlemail.com Fri Aug 20 06:52:37 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 20 Aug 2010 18:52:37 +0800 Subject: [Numpy-discussion] py3k execfile('setup.py') In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 2:07 PM, Fernando Perez wrote: > On Sat, Aug 14, 2010 at 6:11 AM, Ralf Gommers > wrote: > > Installing into a local virtualenv (with either of the methods that work) > > looks good. Into /Library/Frameworks/..., nose refuses to run any tests, > due > > to the files being executable. But I think that's normal. Changing > > permissions to 644 manually it all looks good again. > > You can also pass --exe to nose to override this safety, if you want > (output of nosetests --help below): > > --exe Look for tests in python modules that are > executable. > Normal behavior is to exclude executable modules, > since they may not be import-safe [NOSE_INCLUDE_EXE] > --noexe DO NOT look for tests in python modules that are > executable. (The default on the windows platform is > to > do so.) > > Thanks, that does the trick. Tests run with "np.test(extra_argv=['--exe'])". I didn't notice at first that setuptools always does this, also for python 2.x installs through setupegg.py. Committed in r8655. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From eadrogue at gmx.net Fri Aug 20 08:40:46 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Fri, 20 Aug 2010 14:40:46 +0200 Subject: [Numpy-discussion] lexsort() of datetime objects doesn't work In-Reply-To: References: <20100819224003.GA8722@doriath.local> Message-ID: <20100820124046.GA10201@doriath.local> 19/08/10 @ 18:03 (-0600), thus spake Charles R Harris: > 2010/8/19 Ernest Adrogu? > > > Hi, > > > > I was trying to use lexsort with an array of > > datetime.date objects, but it doesn't seem to work. > > > > In [86]: date = np.array([datetime.date(2000, 9, 17), > > datetime.date(2000, 10, 1), > > datetime.date(2000, 10, 22), > > datetime.date(2000, 11, 1)]) > > > > In [90]: date > > Out[90]: array([2000-09-17, 2000-10-01, 2000-10-22, 2000-11-01], > > dtype=object) > > > > In [91]: np.lexsort((date,)) > > --------------------------------------------------------------------------- > > TypeError Traceback (most recent call last) > > > > /home/ernest/src/footy/dataset.py in () > > ----> 1 > > 2 > > 3 > > 4 > > 5 > > > > TypeError: merge sort not available for item 0 > > > > The normal argsort works, however this is in a method that uses > > lexsort and I can't change it without breaking things. > > > > > The normal argsort calls the clib quicksort with a comparison function > whereas the mergesort needs to be implemented for the datetime data type. It > shouldn't be difficult to do, but it isn't done. I think datetime is > basically int64, so it should be possible to hook things up to call that > sorting routine. > > Please open a ticket for this, it should be fixed. Thank you. Ticket opened here: http://projects.scipy.org/scipy/ticket/1268 Cheers, Ernest From faltet at pytables.org Fri Aug 20 11:27:24 2010 From: faltet at pytables.org (Francesc Alted) Date: Fri, 20 Aug 2010 17:27:24 +0200 Subject: [Numpy-discussion] [ANN] carray: an in-memory compressed data container Message-ID: ====================== Announcing carray 0.1 ====================== What it is ========== carray is a container for numerical data that can be compressed in-memory. The compresion process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. Having data compressed in-memory can reduce the stress of the memory subsystem. The net result is that carray operations can be faster than using a traditional ndarray object from NumPy. What's new ========== Everything ;-) This is the first public release of carray, and it is more a proof of concept to see which possibilities opens efficient compression for in-memory data containers. Currently carray 0.1 implements the basic container creation and enlargeability. Multidimensional arrays are not supported yet, and elements cannot be modified either (i.e. no `__setitem__()` support). Resources ========= Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://github.com/FrancescAlted/carray/downloads Short tutorial: http://github.com/FrancescAlted/carray/blob/master/USAGE.txt Home of Blosc compressor: http://blosc.pytables.org Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- Enjoy! -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From zbyszek at in.waw.pl Fri Aug 20 14:29:41 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Fri, 20 Aug 2010 20:29:41 +0200 Subject: [Numpy-discussion] [ANN] carray: an in-memory compressed data container In-Reply-To: References: Message-ID: <20100820182941.GA23997@in.waw.pl> OK, I've got a case where carray really shines :| zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py numpy 800000 1000 4 1 problem size: (800000) x 1000 = 10^8.90309 time for concat: 4.806s size of the final container: 6103.516 MB zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py concat 800000 1000 4 1 problem size: (800000) x 1000 = 10^8.90309 time for concat: 3.475s size of the final container: 6103.516 MB zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py carray 800000 1000 4 1 problem size: (800000) x 1000 = 10^8.90309 time for concat: 1.434s size of the final container: 373.480 MB Size is set to NOT hit the swap. This is still the easily compressible arange... but still, the results are very nice. Of course when the swap is hit, the ratio between carray and a normal array can grow to infinity :) zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py numpy 1000000 1000 3 1 problem size: (1000000) x 1000 = 10^9 time for concat: 35.700s size of the final container: 7629.395 MB zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py carray 1000000 1000 3 1 problem size: (1000000) x 1000 = 10^9 time for concat: 1.751s size of the final container: 409.633 MB Best, Zbyszek From d.l.goldsmith at gmail.com Fri Aug 20 15:25:28 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 20 Aug 2010 12:25:28 -0700 Subject: [Numpy-discussion] Making MATLAB and Python "play nice" Message-ID: Hi! Please forgive the re-post: I forgot to change the subject line and I haven't seen a response to this yet, so I'm assuming the former might be the cause of the latter. My question follows the quoted posts. Thanks! > From: Sturla Molden > Subject: Re: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John > Hunter) > To: Discussion of Numerical Python > Message-ID: <8C0B2317-2A22-4828-99E8-AC6C0F77824F at molden.no> > Content-Type: text/plain; charset=us-ascii; format=flowed; > delsp=yes > > There are just two sane solutions for Matlab: Either embed CPyton in a > MEX file, or use Matlab's JVM to run Jython ;) > > http://vader.cse.lehigh.edu/~perkins/pymex.html > > Sturla > Date: Wed, 18 Aug 2010 09:33:59 +0100 > From: Robin > Subject: Re: [Numpy-discussion] [SciPy-Dev] Good-bye, sort of (John > Hunter) > To: Discussion of Numerical Python > Message-ID: > > > > Content-Type: text/plain; charset=ISO-8859-1 > Just thought I'd mention another one since this came up: > > http://github.com/kw/pymex > This one works very nicely - it proxies any Python objects so you can > use, should you want to, the Matlab IDE as a python interpreter, > supports numpy arrays etc. Also cross-platform - I even got it to work > with 64 bit matlab/python on windows (in a fork on github). Thanks for the ideas; are any/all of these "solutions" freely/easily redistributable? DG From robert.kern at gmail.com Fri Aug 20 15:30:58 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 20 Aug 2010 14:30:58 -0500 Subject: [Numpy-discussion] Making MATLAB and Python "play nice" In-Reply-To: References: Message-ID: On Fri, Aug 20, 2010 at 14:25, David Goldsmith wrote: > Hi! ?Please forgive the re-post: I forgot to change the subject line > and I haven't seen a response to this yet, so I'm assuming the former > might be the cause of the latter. Or perhaps because the licenses are plainly visible at the links? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From faltet at pytables.org Fri Aug 20 19:31:03 2010 From: faltet at pytables.org (Francesc Alted) Date: Sat, 21 Aug 2010 00:31:03 +0100 Subject: [Numpy-discussion] [ANN] carray: an in-memory compressed data container In-Reply-To: <20100820182941.GA23997@in.waw.pl> References: <20100820182941.GA23997@in.waw.pl> Message-ID: 2010/8/20, Zbyszek Szmek : > OK, I've got a case where carray really shines :| > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > bench/concat.py numpy 800000 1000 4 1 > problem size: (800000) x 1000 = 10^8.90309 > time for concat: 4.806s > size of the final container: 6103.516 MB > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > bench/concat.py concat 800000 1000 4 1 > problem size: (800000) x 1000 = 10^8.90309 > time for concat: 3.475s > size of the final container: 6103.516 MB > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > bench/concat.py carray 800000 1000 4 1 > problem size: (800000) x 1000 = 10^8.90309 > time for concat: 1.434s > size of the final container: 373.480 MB > > Size is set to NOT hit the swap. This is still the easily compressible > arange... but still, the results are very nice. Wow, the results with your processor are much nicer than with my Atom indeed. But yeah, I somewhat expected this because Blosc works much faster with recent processors, as can be seen in: http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks BTW, the difference between memcpy and memmove times for this benchmark is almost 40% for your computer, which is really large :-/ Hmm, something must go really wrong with memcpy in some glibc distributions... At any rate, for real data that is less compressible the advantages of carray will be less apparent, but at least the proof of concept seems to work as intended, so I'm very happy with it. I'm also expecting that the combination carray/numexpr would perform faster than plain computations programmed in C, most specially with modern processors, but will see how much faster exactly. > Of course when the swap is hit, the ratio between carray and a normal array > can grow to infinity :) > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > bench/concat.py numpy 1000000 1000 3 1 > problem size: (1000000) x 1000 = 10^9 > time for concat: 35.700s > size of the final container: 7629.395 MB > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > bench/concat.py carray 1000000 1000 3 1 > problem size: (1000000) x 1000 = 10^9 > time for concat: 1.751s > size of the final container: 409.633 MB Exactly. This is another scenario where the carray concept can be really useful. -- Francesc Alted From thomas.robitaille at gmail.com Sat Aug 21 13:43:55 2010 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sat, 21 Aug 2010 13:43:55 -0400 Subject: [Numpy-discussion] Bug in loadtxt Message-ID: Hi, I am running into a precision issue with np.loadtxt. I have a data file with the following contents: $ cat data.txt -9.9999999961922814E-01 -9.9999999996192290E-01 -9.9999999999619227E-01 -9.9999999999961919E-01 -9.9999999999996192E-01 -9.9999999999999611E-01 -1.0000000000000000E+00 If I try and read this in using loadtxt, which should read numbers in using (64-bit) float by default, I get: Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '2.0.0.dev8657' >>> np.loadtxt('data.txt') array([-1., -1., -1., -1., -1., -1., -1.]) If I now create a file called data2.txt with only the first line: $ cat data2.txt -9.9999999961922814E-01 loadtxt works correctly: Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '2.0.0.dev8657' >>> np.loadtxt('data2.txt') array(-0.99999999961922814) I have submitted a bug report: http://projects.scipy.org/numpy/ticket/1589 Cheers, Tom From josef.pktd at gmail.com Sat Aug 21 13:52:08 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 21 Aug 2010 13:52:08 -0400 Subject: [Numpy-discussion] Bug in loadtxt In-Reply-To: References: Message-ID: On Sat, Aug 21, 2010 at 1:43 PM, Thomas Robitaille wrote: > Hi, > > I am running into a precision issue with np.loadtxt. I have a data file with the following contents: > > $ cat data.txt > -9.9999999961922814E-01 > -9.9999999996192290E-01 > -9.9999999999619227E-01 > -9.9999999999961919E-01 > -9.9999999999996192E-01 > -9.9999999999999611E-01 > -1.0000000000000000E+00 > > If I try and read this in using loadtxt, which should read numbers in using (64-bit) float by default, I get: > > Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) > [GCC 4.2.1 (Apple Inc. build 5646)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy as np >>>> np.__version__ > '2.0.0.dev8657' >>>> np.loadtxt('data.txt') > array([-1., -1., -1., -1., -1., -1., -1.]) > > If I now create a file called data2.txt with only the first line: > > $ cat data2.txt > -9.9999999961922814E-01 > > loadtxt works correctly: > > Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) > [GCC 4.2.1 (Apple Inc. build 5646)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy as np >>>> np.__version__ > '2.0.0.dev8657' >>>> np.loadtxt('data2.txt') > array(-0.99999999961922814) > > I have submitted a bug report: > > http://projects.scipy.org/numpy/ticket/1589 are you sure this is not just a print precision problem? >>> a =np.array([-9.9999999961922814E-01, -9.9999999996192290E-01, -9.9999999999619227E-01, -9.9999999999961919E-01, -9.9999999999996192E-01, -9.9999999999999611E-01, -1.0000000000000000E+00]) >>> a array([-1., -1., -1., -1., -1., -1., -1.]) >>> a[0] -0.99999999961922814 >>> a[:1] array([-1.]) Josef > > Cheers, > > Tom > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From thomas.robitaille at gmail.com Sat Aug 21 14:03:13 2010 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sat, 21 Aug 2010 11:03:13 -0700 (PDT) Subject: [Numpy-discussion] Bug in loadtxt In-Reply-To: References: Message-ID: <29500609.post@talk.nabble.com> josef.pktd wrote: > > are you sure this is not just a print precision problem? > Thanks for pointing this out, it does seem to be just to do with the printing precision. I didn't notice this before, because for the last few elements of the array, print still gives just -1: In [19]: for x in a: ....: print x ....: ....: -0.999999999619 -0.999999999962 -0.999999999996 -1.0 -1.0 -1.0 -1.0 But I now realize that to get the full precision, I should have done In [20]: for x in a: ....: repr(x) ....: ....: Out[20]: '-0.99999999961922814' Out[20]: '-0.9999999999619229' Out[20]: '-0.99999999999619227' Out[20]: '-0.99999999999961919' Out[20]: '-0.99999999999996192' Out[20]: '-0.99999999999999611' Out[20]: '-1.0' Cheers, Tom -- View this message in context: http://old.nabble.com/Bug-in-loadtxt-tp29500522p29500609.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From seb.haase at gmail.com Sat Aug 21 16:34:22 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sat, 21 Aug 2010 22:34:22 +0200 Subject: [Numpy-discussion] [ANN] carray: an in-memory compressed data container In-Reply-To: References: <20100820182941.GA23997@in.waw.pl> Message-ID: Hi Francesc, another exciting project ... congratulations ! Am I correct in thinking that memmapping a carray would also be a great speed advantage over memmapped ndarrays ? Let's say I have a 2Gbyte ndarray memmaped over a NFS network connection, should the speed increase simply scale with the compression factor ? Regards, Sebastian On Sat, Aug 21, 2010 at 1:31 AM, Francesc Alted wrote: > 2010/8/20, Zbyszek Szmek : >> OK, I've got a case where carray really shines :| >> >> zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python >> bench/concat.py numpy 800000 1000 4 1 >> problem size: (800000) x 1000 = 10^8.90309 >> time for concat: 4.806s >> size of the final container: 6103.516 MB >> zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python >> bench/concat.py concat 800000 1000 4 1 >> problem size: (800000) x 1000 = 10^8.90309 >> time for concat: 3.475s >> size of the final container: 6103.516 MB >> zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python >> bench/concat.py carray 800000 1000 4 1 >> problem size: (800000) x 1000 = 10^8.90309 >> time for concat: 1.434s >> size of the final container: 373.480 MB >> >> Size is set to NOT hit the swap. This is still the easily compressible >> arange... but still, the results are very nice. > > Wow, the results with your processor are much nicer than with my Atom > indeed. ?But yeah, I somewhat expected this because Blosc works much > faster with recent processors, as can be seen in: > > http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks > > BTW, the difference between memcpy and memmove times for this > benchmark is almost 40% for your computer, which is really large :-/ > Hmm, something must go really wrong with memcpy in some glibc > distributions... > > At any rate, for real data that is less compressible the advantages of > carray will be less apparent, but at least the proof of concept seems > to work as intended, so I'm very happy with it. ?I'm also expecting > that the combination carray/numexpr would perform faster than plain > computations programmed in C, most specially with modern processors, > but will see how much faster exactly. > >> Of course when the swap is hit, the ratio between carray and a normal array >> can grow to infinity :) >> >> zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python >> bench/concat.py numpy 1000000 1000 3 1 >> problem size: (1000000) x 1000 = 10^9 >> time for concat: 35.700s >> size of the final container: 7629.395 MB >> zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python >> bench/concat.py carray 1000000 1000 3 1 >> problem size: (1000000) x 1000 = 10^9 >> time for concat: 1.751s >> size of the final container: 409.633 MB > > Exactly. ?This is another scenario where the carray concept can be > really useful. > > -- > Francesc Alted > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From seb.haase at gmail.com Sat Aug 21 16:44:11 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sat, 21 Aug 2010 22:44:11 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? Message-ID: Hi, this is somewhat OT for this list, but since I know that David and many others here have lot's of experience compiling C extensions I thought I could just ask: Looking at http://sourceforge.net/projects/mingw-w64/files/ I did not know (even after reading the FAQ) which file to download and how things would eventually work. I have a 64bit windows 7 installed, and got many precompiled packages for amd64 Python 2.7 from http://www.lfd.uci.edu/~gohlke/pythonlibs/ (thanks to Christoph Gohlke for all the work) But now I have some C++ extensions on my own, and know how build them using cygwin -- but that would only produce 32bit modules and should be unusable. So, the question is if someone has or knows of some tutorial about how to go about this - step by step. This info could maybe even go the scipy wiki.... Thanks, Sebastian Haase From cgohlke at uci.edu Sat Aug 21 17:29:08 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 21 Aug 2010 14:29:08 -0700 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: Message-ID: <4C704524.2010303@uci.edu> On 8/21/2010 1:44 PM, Sebastian Haase wrote: > Hi, > > this is somewhat OT for this list, but since I know that David and > many others here have lot's of experience compiling C extensions I > thought I could just ask: > Looking at > http://sourceforge.net/projects/mingw-w64/files/ > I did not know (even after reading the FAQ) which file to download and > how things would eventually work. > > I have a 64bit windows 7 installed, and got many precompiled packages > for amd64 Python 2.7 from > http://www.lfd.uci.edu/~gohlke/pythonlibs/ > (thanks to Christoph Gohlke for all the work) > But now I have some C++ extensions on my own, and know how build them > using cygwin -- but that would only produce 32bit modules and should > be unusable. > > So, the question is if someone has or knows of some tutorial about how > to go about this - step by step. This info could maybe even go the > scipy wiki.... > > Thanks, > Sebastian Haase Hi Sebastian, I am not aware of such a tutorial. There's some information at I did not have good experience last time (about a year ago) I tried mingw-w64. Occasional crashes during compilation and at runtime. Probably that has changed. At least you have to create the missing libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the default crt. You probably know that the "free" Windows 7 Platform SDK can be used to build Python >=2.6 extensions written in C89. -- Christoph From seb.haase at gmail.com Sat Aug 21 17:37:13 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sat, 21 Aug 2010 23:37:13 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: <4C704524.2010303@uci.edu> References: <4C704524.2010303@uci.edu> Message-ID: On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke wrote: > > > On 8/21/2010 1:44 PM, Sebastian Haase wrote: >> Hi, >> >> this is somewhat OT for this list, but since I know that David and >> many others here have lot's of experience compiling C extensions I >> thought I could just ask: >> Looking at >> http://sourceforge.net/projects/mingw-w64/files/ >> I did not know (even after reading the FAQ) which file to download and >> how things would eventually work. >> >> I have a 64bit windows 7 installed, and got many precompiled packages >> for amd64 Python 2.7 from >> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >> (thanks to ?Christoph Gohlke for all the work) >> But now I have some C++ extensions on my own, and know how build them >> using cygwin -- but that would only produce 32bit modules and should >> be unusable. >> >> So, the question is if someone has or knows of some tutorial about how >> to go about this - step by step. This info could maybe even go the >> scipy wiki.... >> >> Thanks, >> Sebastian Haase > > > Hi Sebastian, > > I am not aware of such a tutorial. There's some information at > > > I did not have good experience last time (about a year ago) I tried > mingw-w64. Occasional crashes during compilation and at runtime. > Probably that has changed. At least you have to create the missing > libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the > default crt. > > You probably know that the "free" Windows 7 Platform SDK can be used to > build Python >=2.6 extensions written in C89. > > > -- Hi Christoph, I did not exactly know this - thanks for the info (I knew about something called Visual Studio Express 2003- but that only works/worked for Python 2.5, I think...) Rephrasing my original question: Is the mingw-w64 at all "easy" by now ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that I could easily run on virtualbox) ? (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so I'll look into that -- giant download !?) Thanks, Sebastian From cgohlke at uci.edu Sat Aug 21 18:02:21 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 21 Aug 2010 15:02:21 -0700 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> Message-ID: <4C704CED.2030602@uci.edu> On 8/21/2010 2:37 PM, Sebastian Haase wrote: > On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke wrote: >> >> >> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>> Hi, >>> >>> this is somewhat OT for this list, but since I know that David and >>> many others here have lot's of experience compiling C extensions I >>> thought I could just ask: >>> Looking at >>> http://sourceforge.net/projects/mingw-w64/files/ >>> I did not know (even after reading the FAQ) which file to download and >>> how things would eventually work. >>> >>> I have a 64bit windows 7 installed, and got many precompiled packages >>> for amd64 Python 2.7 from >>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>> (thanks to Christoph Gohlke for all the work) >>> But now I have some C++ extensions on my own, and know how build them >>> using cygwin -- but that would only produce 32bit modules and should >>> be unusable. >>> >>> So, the question is if someone has or knows of some tutorial about how >>> to go about this - step by step. This info could maybe even go the >>> scipy wiki.... >>> >>> Thanks, >>> Sebastian Haase >> >> >> Hi Sebastian, >> >> I am not aware of such a tutorial. There's some information at >> >> >> I did not have good experience last time (about a year ago) I tried >> mingw-w64. Occasional crashes during compilation and at runtime. >> Probably that has changed. At least you have to create the missing >> libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the >> default crt. >> >> You probably know that the "free" Windows 7 Platform SDK can be used to >> build Python>=2.6 extensions written in C89. >> >> >> -- > Hi Christoph, > > I did not exactly know this - thanks for the info (I knew about > something called Visual Studio Express 2003- but that only > works/worked for Python 2.5, I think...) You can use Visual Studio Express 2008 for building 32 bit extensions for Python >=2.6. > > Rephrasing my original question: Is the mingw-w64 at all "easy" by now Don't know. David Cournapeau probably has the most experience. http://bugs.python.org/issue4709 http://www.equation.com/servlet/equation.cmd?call=fortran > ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that > I could easily run on virtualbox) ? I am also interested in cross compiling on Ubuntu but have not found the time to get started. The IOCBio project cross-compiles their 32 bit extensions on Linux . But as you can see they use Wine and Mingw... > > (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so > I'll look into that -- giant download !?) About one GB. > > Thanks, > Sebastian -- Christoph From d.l.goldsmith at gmail.com Sat Aug 21 18:58:59 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sat, 21 Aug 2010 15:58:59 -0700 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 47, Issue 61 In-Reply-To: References: Message-ID: On Sat, Aug 21, 2010 at 10:00 AM, wrote: > Date: Fri, 20 Aug 2010 14:30:58 -0500 > From: Robert Kern > Subject: Re: [Numpy-discussion] Making MATLAB and Python "play nice" > To: Discussion of Numerical Python > Message-ID: > > > > Content-Type: text/plain; charset=UTF-8 > > On Fri, Aug 20, 2010 at 14:25, David Goldsmith > wrote: > > Hi! ?Please forgive the re-post: I forgot to change the subject line > > and I haven't seen a response to this yet, so I'm assuming the former > > might be the cause of the latter. > > Or perhaps because the licenses are plainly visible at the links? > Ah, I see: if I can't be bothered to click on the links, no one else can be bothered to tell me that that's all I need to do to get my question answered. Unfortunately, the solutions are useless to me if they're not freely redistributable, so I have no incentive to click on the links--which do not advertise that they answer the licensing question--'til that question is answered - catch-22. DG > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > > > ------------------------------ > > Message: 4 > Date: Sat, 21 Aug 2010 00:31:03 +0100 > From: Francesc Alted > Subject: Re: [Numpy-discussion] [ANN] carray: an in-memory compressed > data container > To: Discussion of Numerical Python > Message-ID: > > > > Content-Type: text/plain; charset=ISO-8859-1 > > 2010/8/20, Zbyszek Szmek : > > OK, I've got a case where carray really shines :| > > > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py numpy 800000 1000 4 1 > > problem size: (800000) x 1000 = 10^8.90309 > > time for concat: 4.806s > > size of the final container: 6103.516 MB > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py concat 800000 1000 4 1 > > problem size: (800000) x 1000 = 10^8.90309 > > time for concat: 3.475s > > size of the final container: 6103.516 MB > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py carray 800000 1000 4 1 > > problem size: (800000) x 1000 = 10^8.90309 > > time for concat: 1.434s > > size of the final container: 373.480 MB > > > > Size is set to NOT hit the swap. This is still the easily compressible > > arange... but still, the results are very nice. > > Wow, the results with your processor are much nicer than with my Atom > indeed. But yeah, I somewhat expected this because Blosc works much > faster with recent processors, as can be seen in: > > http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks > > BTW, the difference between memcpy and memmove times for this > benchmark is almost 40% for your computer, which is really large :-/ > Hmm, something must go really wrong with memcpy in some glibc > distributions... > > At any rate, for real data that is less compressible the advantages of > carray will be less apparent, but at least the proof of concept seems > to work as intended, so I'm very happy with it. I'm also expecting > that the combination carray/numexpr would perform faster than plain > computations programmed in C, most specially with modern processors, > but will see how much faster exactly. > > > Of course when the swap is hit, the ratio between carray and a normal > array > > can grow to infinity :) > > > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py numpy 1000000 1000 3 1 > > problem size: (1000000) x 1000 = 10^9 > > time for concat: 35.700s > > size of the final container: 7629.395 MB > > zbyszek at escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py carray 1000000 1000 3 1 > > problem size: (1000000) x 1000 = 10^9 > > time for concat: 1.751s > > size of the final container: 409.633 MB > > Exactly. This is another scenario where the carray concept can be > really useful. > > -- > Francesc Alted > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 47, Issue 61 > ************************************************ > -- Privacy is overrated; Identity isn't. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Aug 21 19:12:50 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 21 Aug 2010 18:12:50 -0500 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 47, Issue 61 In-Reply-To: References: Message-ID: On Sat, Aug 21, 2010 at 17:58, David Goldsmith wrote: > On Sat, Aug 21, 2010 at 10:00 AM, > wrote: >> >> Date: Fri, 20 Aug 2010 14:30:58 -0500 >> From: Robert Kern >> Subject: Re: [Numpy-discussion] Making MATLAB and Python "play nice" >> To: Discussion of Numerical Python >> Message-ID: >> ? ? ? ? >> Content-Type: text/plain; charset=UTF-8 >> >> On Fri, Aug 20, 2010 at 14:25, David Goldsmith >> wrote: >> > Hi! ?Please forgive the re-post: I forgot to change the subject line >> > and I haven't seen a response to this yet, so I'm assuming the former >> > might be the cause of the latter. >> >> Or perhaps because the licenses are plainly visible at the links? > > Ah, I see: if I can't be bothered to click on the links, no one else can be > bothered to tell me that that's all I need to do to get my question > answered. Well, the real reason you didn't get a response is most likely the usual: no one got around to it, or lost track of it. Though responding to the digest is not helping you now. > Unfortunately, the solutions are useless to me if they're not > freely redistributable, so I have no incentive to click on the links--which > do not advertise that they answer the licensing question-- It's pretty standard these days for home pages of software to contain the license information. No one needs to advertise that any more. > 'til that question > is answered - catch-22. Do you really think that "actually solving the problem you posed" is not enough of an incentive to do the minimal action of clicking a link and reading a little? Hell, answering the question "Are they redistributable?" is incentive to click the links. It's faster and easier than posting to the list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From faltet at pytables.org Sat Aug 21 19:29:41 2010 From: faltet at pytables.org (Francesc Alted) Date: Sun, 22 Aug 2010 00:29:41 +0100 Subject: [Numpy-discussion] [ANN] carray: an in-memory compressed data container In-Reply-To: References: <20100820182941.GA23997@in.waw.pl> Message-ID: Hey Sebastian, 2010/8/21, Sebastian Haase : > Hi Francesc, > > another exciting project ... congratulations ! Thanks! > Am I correct in thinking that memmapping a carray would also be a > great speed advantage over memmapped ndarrays ? Let's say I have a > 2Gbyte ndarray memmaped over a NFS network connection, should the > speed increase simply scale with the compression factor ? Mmh, in principle yes. However, carray is based on the concept of independent chunks of data and frankly, it does not make a lot of sense to me having to create many small memmapped files in order to keep the chunks. Instead, I'd use PyTables (what else? ;-) for this because it is also based on the same chunk concept than carray, but chunks are saved on a monolithic (HDF5) file, which is much easier to handle. These chunks can be compressed with Blosc too, so I/O is fast (although due to the HDF5 overhead, probably a compressed memmap approach might be faster yet, but much more difficult to manage). And last but not least, this does not have the limitation of virtual memory size of memmaped solutions, which I find quite uncomfortable. -- Francesc Alted From seb.haase at gmail.com Sat Aug 21 21:39:45 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 22 Aug 2010 03:39:45 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: <4C704CED.2030602@uci.edu> References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 12:02 AM, Christoph Gohlke wrote: > > > On 8/21/2010 2:37 PM, Sebastian Haase wrote: >> On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke ?wrote: >>> >>> >>> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>>> Hi, >>>> >>>> this is somewhat OT for this list, but since I know that David and >>>> many others here have lot's of experience compiling C extensions I >>>> thought I could just ask: >>>> Looking at >>>> http://sourceforge.net/projects/mingw-w64/files/ >>>> I did not know (even after reading the FAQ) which file to download and >>>> how things would eventually work. >>>> >>>> I have a 64bit windows 7 installed, and got many precompiled packages >>>> for amd64 Python 2.7 from >>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>>> (thanks to ?Christoph Gohlke for all the work) >>>> But now I have some C++ extensions on my own, and know how build them >>>> using cygwin -- but that would only produce 32bit modules and should >>>> be unusable. >>>> >>>> So, the question is if someone has or knows of some tutorial about how >>>> to go about this - step by step. This info could maybe even go the >>>> scipy wiki.... >>>> >>>> Thanks, >>>> Sebastian Haase >>> >>> >>> Hi Sebastian, >>> >>> I am not aware of such a tutorial. There's some information at >>> >>> >>> I did not have good experience last time (about a year ago) I tried >>> mingw-w64. Occasional crashes during compilation and at runtime. >>> Probably that has changed. At least you have to create the missing >>> libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the >>> default crt. >>> >>> You probably know that the "free" Windows 7 Platform SDK can be used to >>> build Python>=2.6 extensions written in C89. >>> >>> >>> -- >> Hi Christoph, >> >> I did not exactly know this - thanks for the info (I knew about >> something called Visual Studio Express 2003- but that only >> works/worked for Python 2.5, I think...) > > You can use Visual Studio Express 2008 for building 32 bit extensions > for Python >=2.6. > >> >> Rephrasing my original question: Is the mingw-w64 at all "easy" by now > > Don't know. David Cournapeau probably has the most experience. > > http://bugs.python.org/issue4709 > http://www.equation.com/servlet/equation.cmd?call=fortran > >> ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that >> I could easily run on virtualbox) ? > > I am also interested in cross compiling on Ubuntu but have not found the > time to get started. The IOCBio project cross-compiles their 32 bit > extensions on Linux > . > But as you can see they use Wine and Mingw... > >> >> (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so >> I'll look into that -- giant download !?) > > About one GB. > >> Do you know if that contains a C++ compiler ? The first page before it starts the actual download has "Visual C++ Compilers" grayed out ... !? -Sebastian From kwatford+scipy at gmail.com Sun Aug 22 00:27:36 2010 From: kwatford+scipy at gmail.com (Ken Watford) Date: Sun, 22 Aug 2010 00:27:36 -0400 Subject: [Numpy-discussion] Making MATLAB and Python "play nice" In-Reply-To: References: Message-ID: On Fri, Aug 20, 2010 at 3:25 PM, David Goldsmith wrote: > Hi! ?Please forgive the re-post: I forgot to change the subject line > and I haven't seen a response to this yet, so I'm assuming the former > might be the cause of the latter. ?My question follows the quoted > posts. ?Thanks! > > *snip* > > Thanks for the ideas; are any/all of these "solutions" freely/easily > redistributable? > > DG The one Robin suggested is mine. It uses the MIT license, so basically you can do whatever you want with it. I haven't really been maintaining it, as I've been busy with other things, though I have been pondering a rewrite in Cython. If someone else wants to take charge of it, though, feel free. (It's amusing that both are called pymex. I spoke with the other guy once, and apparently we picked names at around the same time. I guess for what it is, that's the most obvious name one could pick.) From seb.haase at gmail.com Sun Aug 22 02:16:57 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 22 Aug 2010 08:16:57 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 3:39 AM, Sebastian Haase wrote: > On Sun, Aug 22, 2010 at 12:02 AM, Christoph Gohlke wrote: >> >> >> On 8/21/2010 2:37 PM, Sebastian Haase wrote: >>> On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke ?wrote: >>>> >>>> >>>> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>>>> Hi, >>>>> >>>>> this is somewhat OT for this list, but since I know that David and >>>>> many others here have lot's of experience compiling C extensions I >>>>> thought I could just ask: >>>>> Looking at >>>>> http://sourceforge.net/projects/mingw-w64/files/ >>>>> I did not know (even after reading the FAQ) which file to download and >>>>> how things would eventually work. >>>>> >>>>> I have a 64bit windows 7 installed, and got many precompiled packages >>>>> for amd64 Python 2.7 from >>>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>>>> (thanks to ?Christoph Gohlke for all the work) >>>>> But now I have some C++ extensions on my own, and know how build them >>>>> using cygwin -- but that would only produce 32bit modules and should >>>>> be unusable. >>>>> >>>>> So, the question is if someone has or knows of some tutorial about how >>>>> to go about this - step by step. This info could maybe even go the >>>>> scipy wiki.... >>>>> >>>>> Thanks, >>>>> Sebastian Haase >>>> >>>> >>>> Hi Sebastian, >>>> >>>> I am not aware of such a tutorial. There's some information at >>>> >>>> >>>> I did not have good experience last time (about a year ago) I tried >>>> mingw-w64. Occasional crashes during compilation and at runtime. >>>> Probably that has changed. At least you have to create the missing >>>> libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the >>>> default crt. >>>> >>>> You probably know that the "free" Windows 7 Platform SDK can be used to >>>> build Python>=2.6 extensions written in C89. >>>> >>>> >>>> -- >>> Hi Christoph, >>> >>> I did not exactly know this - thanks for the info (I knew about >>> something called Visual Studio Express 2003- but that only >>> works/worked for Python 2.5, I think...) >> >> You can use Visual Studio Express 2008 for building 32 bit extensions >> for Python >=2.6. >> >>> >>> Rephrasing my original question: Is the mingw-w64 at all "easy" by now >> >> Don't know. David Cournapeau probably has the most experience. >> >> http://bugs.python.org/issue4709 >> http://www.equation.com/servlet/equation.cmd?call=fortran >> >>> ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that >>> I could easily run on virtualbox) ? >> >> I am also interested in cross compiling on Ubuntu but have not found the >> time to get started. The IOCBio project cross-compiles their 32 bit >> extensions on Linux >> . >> But as you can see they use Wine and Mingw... >> >>> >>> (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so >>> I'll look into that -- giant download !?) >> >> About one GB. >> >>> > Do you know if that contains a C++ compiler ? ?The first page before > it starts the actual download has "Visual C++ Compilers" grayed out > ... !? > > -Sebastian > Ok, apparently I had to install the "dot NET Framework 4" from http://msdn.microsoft.com/en-us/netframework/aa569263.aspx first, before then the C++ could be installed. But now setup.py still complains: error: Unable to find vcvarsall.bat and I think it is looking for C:\Program Files (x86)\Microsoft Visual Studio 9.0 while that file got installed in C:\Program Files (x86)\Microsoft Visual Studio 10.0 I don't know how to get the "log.debug" messages from the setup.py script activated... ? Still trying .... Cheers, Sebastian From seb.haase at gmail.com Sun Aug 22 02:40:38 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 22 Aug 2010 08:40:38 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 8:16 AM, Sebastian Haase wrote: > On Sun, Aug 22, 2010 at 3:39 AM, Sebastian Haase wrote: >> On Sun, Aug 22, 2010 at 12:02 AM, Christoph Gohlke wrote: >>> >>> >>> On 8/21/2010 2:37 PM, Sebastian Haase wrote: >>>> On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke ?wrote: >>>>> >>>>> >>>>> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>>>>> Hi, >>>>>> >>>>>> this is somewhat OT for this list, but since I know that David and >>>>>> many others here have lot's of experience compiling C extensions I >>>>>> thought I could just ask: >>>>>> Looking at >>>>>> http://sourceforge.net/projects/mingw-w64/files/ >>>>>> I did not know (even after reading the FAQ) which file to download and >>>>>> how things would eventually work. >>>>>> >>>>>> I have a 64bit windows 7 installed, and got many precompiled packages >>>>>> for amd64 Python 2.7 from >>>>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>>>>> (thanks to ?Christoph Gohlke for all the work) >>>>>> But now I have some C++ extensions on my own, and know how build them >>>>>> using cygwin -- but that would only produce 32bit modules and should >>>>>> be unusable. >>>>>> >>>>>> So, the question is if someone has or knows of some tutorial about how >>>>>> to go about this - step by step. This info could maybe even go the >>>>>> scipy wiki.... >>>>>> >>>>>> Thanks, >>>>>> Sebastian Haase >>>>> >>>>> >>>>> Hi Sebastian, >>>>> >>>>> I am not aware of such a tutorial. There's some information at >>>>> >>>>> >>>>> I did not have good experience last time (about a year ago) I tried >>>>> mingw-w64. Occasional crashes during compilation and at runtime. >>>>> Probably that has changed. At least you have to create the missing >>>>> libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the >>>>> default crt. >>>>> >>>>> You probably know that the "free" Windows 7 Platform SDK can be used to >>>>> build Python>=2.6 extensions written in C89. >>>>> >>>>> >>>>> -- >>>> Hi Christoph, >>>> >>>> I did not exactly know this - thanks for the info (I knew about >>>> something called Visual Studio Express 2003- but that only >>>> works/worked for Python 2.5, I think...) >>> >>> You can use Visual Studio Express 2008 for building 32 bit extensions >>> for Python >=2.6. >>> >>>> >>>> Rephrasing my original question: Is the mingw-w64 at all "easy" by now >>> >>> Don't know. David Cournapeau probably has the most experience. >>> >>> http://bugs.python.org/issue4709 >>> http://www.equation.com/servlet/equation.cmd?call=fortran >>> >>>> ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that >>>> I could easily run on virtualbox) ? >>> >>> I am also interested in cross compiling on Ubuntu but have not found the >>> time to get started. The IOCBio project cross-compiles their 32 bit >>> extensions on Linux >>> . >>> But as you can see they use Wine and Mingw... >>> >>>> >>>> (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so >>>> I'll look into that -- giant download !?) >>> >>> About one GB. >>> >>>> >> Do you know if that contains a C++ compiler ? ?The first page before >> it starts the actual download has "Visual C++ Compilers" grayed out >> ... !? >> >> -Sebastian >> > Ok, apparently I had to install the "dot NET Framework 4" from > http://msdn.microsoft.com/en-us/netframework/aa569263.aspx > first, before then the C++ could be installed. > But now setup.py still complains: > ? ? ? error: Unable to find vcvarsall.bat > and I think it is looking for > C:\Program Files (x86)\Microsoft Visual Studio 9.0 > while that file got installed in > C:\Program Files (x86)\Microsoft Visual Studio 10.0 > I don't know how to get the "log.debug" messages from the setup.py > script activated... > ? I got the debug out by adding a "-d" \Python27\python.exe setup.py -v build (even though setup.py --help states that --verbose should be already the default, well...) Then I mentioned the environment variable %VS90COMNTOOLS% which I then set to $$>set VS90COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin But now I'm getting a traceback: File "C:\Python27\lib\distutils\msvc9compiler.py", line 469, in compile self.initialize() File "C:\Python27\lib\distutils\msvc9compiler.py", line 379, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python27\lib\distutils\msvc9compiler.py", line 295, in query_vcvarsall raise ValueError(str(list(result.keys()))) ValueError: [u'path'] Well, still trying .... Cheers, Sebastian From seb.haase at gmail.com Sun Aug 22 03:41:39 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 22 Aug 2010 09:41:39 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 8:40 AM, Sebastian Haase wrote: > On Sun, Aug 22, 2010 at 8:16 AM, Sebastian Haase wrote: >> On Sun, Aug 22, 2010 at 3:39 AM, Sebastian Haase wrote: >>> On Sun, Aug 22, 2010 at 12:02 AM, Christoph Gohlke wrote: >>>> >>>> >>>> On 8/21/2010 2:37 PM, Sebastian Haase wrote: >>>>> On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke ?wrote: >>>>>> >>>>>> >>>>>> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>>>>>> Hi, >>>>>>> >>>>>>> this is somewhat OT for this list, but since I know that David and >>>>>>> many others here have lot's of experience compiling C extensions I >>>>>>> thought I could just ask: >>>>>>> Looking at >>>>>>> http://sourceforge.net/projects/mingw-w64/files/ >>>>>>> I did not know (even after reading the FAQ) which file to download and >>>>>>> how things would eventually work. >>>>>>> >>>>>>> I have a 64bit windows 7 installed, and got many precompiled packages >>>>>>> for amd64 Python 2.7 from >>>>>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>>>>>> (thanks to ?Christoph Gohlke for all the work) >>>>>>> But now I have some C++ extensions on my own, and know how build them >>>>>>> using cygwin -- but that would only produce 32bit modules and should >>>>>>> be unusable. >>>>>>> >>>>>>> So, the question is if someone has or knows of some tutorial about how >>>>>>> to go about this - step by step. This info could maybe even go the >>>>>>> scipy wiki.... >>>>>>> >>>>>>> Thanks, >>>>>>> Sebastian Haase >>>>>> >>>>>> >>>>>> Hi Sebastian, >>>>>> >>>>>> I am not aware of such a tutorial. There's some information at >>>>>> >>>>>> >>>>>> I did not have good experience last time (about a year ago) I tried >>>>>> mingw-w64. Occasional crashes during compilation and at runtime. >>>>>> Probably that has changed. At least you have to create the missing >>>>>> libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the >>>>>> default crt. >>>>>> >>>>>> You probably know that the "free" Windows 7 Platform SDK can be used to >>>>>> build Python>=2.6 extensions written in C89. >>>>>> >>>>>> >>>>>> -- >>>>> Hi Christoph, >>>>> >>>>> I did not exactly know this - thanks for the info (I knew about >>>>> something called Visual Studio Express 2003- but that only >>>>> works/worked for Python 2.5, I think...) >>>> >>>> You can use Visual Studio Express 2008 for building 32 bit extensions >>>> for Python >=2.6. >>>> >>>>> >>>>> Rephrasing my original question: Is the mingw-w64 at all "easy" by now >>>> >>>> Don't know. David Cournapeau probably has the most experience. >>>> >>>> http://bugs.python.org/issue4709 >>>> http://www.equation.com/servlet/equation.cmd?call=fortran >>>> >>>>> ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that >>>>> I could easily run on virtualbox) ? >>>> >>>> I am also interested in cross compiling on Ubuntu but have not found the >>>> time to get started. The IOCBio project cross-compiles their 32 bit >>>> extensions on Linux >>>> . >>>> But as you can see they use Wine and Mingw... >>>> >>>>> >>>>> (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so >>>>> I'll look into that -- giant download !?) >>>> >>>> About one GB. >>>> >>>>> >>> Do you know if that contains a C++ compiler ? ?The first page before >>> it starts the actual download has "Visual C++ Compilers" grayed out >>> ... !? >>> >>> -Sebastian >>> >> Ok, apparently I had to install the "dot NET Framework 4" from >> http://msdn.microsoft.com/en-us/netframework/aa569263.aspx >> first, before then the C++ could be installed. >> But now setup.py still complains: >> ? ? ? error: Unable to find vcvarsall.bat >> and I think it is looking for >> C:\Program Files (x86)\Microsoft Visual Studio 9.0 >> while that file got installed in >> C:\Program Files (x86)\Microsoft Visual Studio 10.0 >> I don't know how to get the "log.debug" messages from the setup.py >> script activated... >> ? > > I got the debug out by adding a "-d" > \Python27\python.exe setup.py -v build > (even though setup.py --help ?states that --verbose should be already > the default, well...) > > Then I mentioned the environment variable %VS90COMNTOOLS% > which I then set to > $$>set VS90COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin > > But now I'm getting a traceback: > ?File "C:\Python27\lib\distutils\msvc9compiler.py", line 469, in compile > ? ?self.initialize() > ?File "C:\Python27\lib\distutils\msvc9compiler.py", line 379, in initialize > ? ?vc_env = query_vcvarsall(VERSION, plat_spec) > ?File "C:\Python27\lib\distutils\msvc9compiler.py", line 295, in > query_vcvarsall > ? ?raise ValueError(str(list(result.keys()))) > ValueError: [u'path'] > > Well, still trying .... > Cheers, > Sebastian > Now, I renamed the file C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat (as mentioned in http://mattptr.net/2010/07/28/building-python-extensions-in-a-modern-windows-environment/) to vcvars64.bat and moved it to C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64 The compiler races through and I end up getting linker problems: LINK : error LNK2001: unresolved external symbol init_sebmodule We'll see, Sebastian From robince at gmail.com Sun Aug 22 04:35:14 2010 From: robince at gmail.com (Robin) Date: Sun, 22 Aug 2010 09:35:14 +0100 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 8:41 AM, Sebastian Haase wrote: >>>> Do you know if that contains a C++ compiler ? ?The first page before >>>> it starts the actual download has "Visual C++ Compilers" grayed out >>>> ... !? >>>> >>>> -Sebastian >>>> >>> Ok, apparently I had to install the "dot NET Framework 4" from >>> http://msdn.microsoft.com/en-us/netframework/aa569263.aspx >>> first, before then the C++ could be installed. >>> But now setup.py still complains: >>> ? ? ? error: Unable to find vcvarsall.bat >>> and I think it is looking for >>> C:\Program Files (x86)\Microsoft Visual Studio 9.0 >>> while that file got installed in >>> C:\Program Files (x86)\Microsoft Visual Studio 10.0 >>> I don't know how to get the "log.debug" messages from the setup.py >>> script activated... >>> ? My limited understanding is that there are two different versions of microsoft stuff at the moment: VS 2008 which goes with SDK 3.5 and VS 2010 which goes with SDK 4.0. I think Python is built with 2008 so that might be easier to try. I'm not sure though. While the SDK contains some compilers I think the easiest way to get the compilers working is to first install the VS express edition, which is 32 bit, and then install the SDK which provides 64 bit support. Here are some instructions I lifted from the Matlab website (I haven't tried building Python extensions yet but I was able to embed Python with this setup). To install Visual Studio 2008 Express Edition with all required components: 1. Install Microsoft Visual Studio 2008 Express Edition. The main Visual Studio 2008 Express installer is available from: http://www.microsoft.com/express/Downloads/#Visual_Studio_2008_Express_Downloads This package can be installed using the default options. 2. Install the Microsoft Windows SDK. The Microsoft Windows SDK is available by searching Microsoft's download site, or by going directly to: http://msdn.microsoft.com/en-us/windowsserver/bb980924.aspx or http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en Download the Windows Server 2008 & .NET 3.5 SDK. Do not install beta or 'Release Candidate' (RC) versions. Also do NOT install "Microsoft Windows SDK for Windows 7 and .NET Framework 4" (version 7.1); if you want to use a 7.x version choose the "Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1". 2.1. While installing the SDK, you must select "x64 Compilers and Tools". For example, in the SDK installer above: On screen "Installation Options" Select "Developer Tools"->"Visual C++ Compilers". This item has the Feature Description "Install the Visual C++ 9.0 Compilers. These compilers allow you to target x86, x64, IA64 processor architectures." 3. To verify that you have all installed components, check that the Microsoft SDK contains the "amd64" version of the C/C++ compiler "cl.exe". This is usually installed into C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\cl.exe Cheers Robin From sturla at molden.no Sun Aug 22 08:58:25 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 22 Aug 2010 14:58:25 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: <4C704CED.2030602@uci.edu> References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: Remember that you will need the Win 7 SDK for .NET 3.5. The version for .NET 4 links with the wrong CRT. It's 1 GB download so Pick the correct. Remember that Python distutils needs the environment variables DISTUTILS_USE_SDK and MSSdk set to use it. The first we must set manually, the latter comes from running setenv. VS2008 Express has a 64 bit compiler. It's not available from the IDE, but we don't need that for Python extensions. Sturla Sendt fra min iPhone Den 22. aug. 2010 kl. 00.02 skrev Christoph Gohlke : > > > On 8/21/2010 2:37 PM, Sebastian Haase wrote: >> On Sat, Aug 21, 2010 at 11:29 PM, Christoph >> Gohlke wrote: >>> >>> >>> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>>> Hi, >>>> >>>> this is somewhat OT for this list, but since I know that David and >>>> many others here have lot's of experience compiling C extensions I >>>> thought I could just ask: >>>> Looking at >>>> http://sourceforge.net/projects/mingw-w64/files/ >>>> I did not know (even after reading the FAQ) which file to >>>> download and >>>> how things would eventually work. >>>> >>>> I have a 64bit windows 7 installed, and got many precompiled >>>> packages >>>> for amd64 Python 2.7 from >>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>>> (thanks to Christoph Gohlke for all the work) >>>> But now I have some C++ extensions on my own, and know how build >>>> them >>>> using cygwin -- but that would only produce 32bit modules and >>>> should >>>> be unusable. >>>> >>>> So, the question is if someone has or knows of some tutorial >>>> about how >>>> to go about this - step by step. This info could maybe even go the >>>> scipy wiki.... >>>> >>>> Thanks, >>>> Sebastian Haase >>> >>> >>> Hi Sebastian, >>> >>> I am not aware of such a tutorial. There's some information at >>> >>> >>> I did not have good experience last time (about a year ago) I tried >>> mingw-w64. Occasional crashes during compilation and at runtime. >>> Probably that has changed. At least you have to create the missing >>> libpython and libmsvcr90 libraries from the dlls and make >>> libmsvcr90 the >>> default crt. >>> >>> You probably know that the "free" Windows 7 Platform SDK can be >>> used to >>> build Python>=2.6 extensions written in C89. >>> >> > >>> >>> -- >> Hi Christoph, >> >> I did not exactly know this - thanks for the info (I knew about >> something called Visual Studio Express 2003- but that only >> works/worked for Python 2.5, I think...) > > You can use Visual Studio Express 2008 for building 32 bit extensions > for Python >=2.6. > >> >> Rephrasing my original question: Is the mingw-w64 at all "easy" by >> now > > Don't know. David Cournapeau probably has the most experience. > > http://bugs.python.org/issue4709 > http://www.equation.com/servlet/equation.cmd?call=fortran > >> ? How about cross compiling to 64bit Windows from a 32bit Ubuntu >> (that >> I could easily run on virtualbox) ? > > I am also interested in cross compiling on Ubuntu but have not found > the > time to get started. The IOCBio project cross-compiles their 32 bit > extensions on Linux > . > But as you can see they use Wine and Mingw... > >> >> (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so >> I'll look into that -- giant download !?) > > About one GB. > >> >> Thanks, >> Sebastian > > -- > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From njs at pobox.com Sun Aug 22 17:36:47 2010 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 22 Aug 2010 14:36:47 -0700 Subject: [Numpy-discussion] questions about a "complicated" user-defined dtype and the ufunc API Message-ID: I'm experimenting with a user-defined "enumeration" dtype -- where the underlying array holds a set of integers, but they (mostly) appear to the user as strings. (This would be potentially useful for representing categorical data, modeling hdf5 enumerations, etc.) So for each set of enumerated values, I have one Python-level object that stores the mapping between strings and integers (an instance of the 'Enum' class), and a few Python-level objects that represent each of the enumerated values (instances of the 'EnumValue' class). To map this into numpy, I've defined and registered a single custom dtype with 'EnumValue' as its typeobj, and then when I want to create an array of enumerations I make a copy of this registered dtype (with PyArray_DescrNewFromType) and stash a reference to the appropriate Enum object in its 'metadata' dictionary. Question 1: Is this overall approach -- of only calling PyArray_RegisterDataType once, and then sharing the resulting typenum among all my different dtype instances -- correct? It doesn't seem reasonable to register a new dtype for every different set of enumerations, because AFAICT this would create a memory leak (since you can't "unregister" a dtype). Anyway, that seems to be working okay, but then I wanted to teach "==" about my new dtype, so that I can compare to strings and such. So, I need to register my comparison function with the "np.equal" ufunc. Question 2: Am I missing something, or does the ufunc API make this impossible? The problem is that a "PyUFuncGenericFunction" doesn't have any way to find the dtypes of the arrays that it's working on. All of the PyArray_ArrFuncs functions take a pointer to the underlying ndarray as an argument, so that when working with a string or void array, you can find the actual dtype and figure out (e.g.) the size of the objects involved. But ufunc inner loops don't get that, so I guess it's just impossible to define a ufunc over variable-sized data, or data that you need to be able to see the dtype metadata to interpret? This seems easy enough to fix, and would probably allow the removal of a big pile of code in arrayobject.c that does special-case handling for "==" on strings and void arrays. (Another side-effect of the current special-case approach is that "==" and "np.equal" behave differently on arrays of strings.) But is there another option? Anyway, thanks for your help! I'll attach the code in case it helps (and because I'm not too confident I'm getting the details right!). Sample usage: >>> from npenum import *; import numpy as np >>> e = Enum(["low", "medium", "high"]) >>> a = np.array(["low", "low", "high", "medium"], dtype=e) >>> a array([low, low, high, medium], dtype=EnumValue) >>> a.view(Enum.get(a).inttype) array([0, 0, 2, 1], dtype=uint32) Thanks, -- Nathaniel -------------- next part -------------- A non-text attachment was scrubbed... Name: npenum.pyx Type: application/octet-stream Size: 12198 bytes Desc: not available URL: From martin.djokovic at gmail.com Sun Aug 22 18:29:27 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Sun, 22 Aug 2010 18:29:27 -0400 Subject: [Numpy-discussion] numpy installation problems Message-ID: Hello everyone, I have an old machine with Fedora 6 installed -numpy was installed but when I tried to install scipy it did not work so I gave up but that messed up my already installed numpy and now I get the following error message ******************************************** >>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 44, in ? import fft File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? from fftpack import * File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? import fftpack_lite as fftpack ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 ********************************************** I am not very conversant with Linux systems but from googling I am guessing this is to do with the C compiler-sould I reinstall something? Your help would be much appreciated. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Aug 22 18:38:52 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 23 Aug 2010 07:38:52 +0900 Subject: [Numpy-discussion] numpy installation problems In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 7:29 AM, martin djokovic wrote: > Hello everyone, > > I have an old machine with Fedora 6 installed -numpy was installed but when > I tried to install scipy it did not work so I gave up but that messed up my > already installed numpy and now I get the following error message Looks like numpy was built with a special compiler (maybe Intel compiler ?). If so, you need to use the same one to build scipy, cheers, David From ben.root at ou.edu Sun Aug 22 18:41:24 2010 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 22 Aug 2010 17:41:24 -0500 Subject: [Numpy-discussion] numpy installation problems In-Reply-To: References: Message-ID: On Sun, Aug 22, 2010 at 5:29 PM, martin djokovic wrote: > Hello everyone, > > I have an old machine with Fedora 6 installed -numpy was installed but when > I tried to install scipy it did not work so I gave up but that messed up my > already installed numpy and now I get the following error message > ******************************************** > >>> import numpy > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 44, > in ? > import fft > File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line > 4, in ? > from fftpack import * > File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line > 28, in ? > import fftpack_lite as fftpack > ImportError: > /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined > symbol: vmldCos2 > ********************************************** > > I am not very conversant with Linux systems but from googling I am guessing > this is to do with the C compiler-sould I reinstall something? > Your help would be much appreciated. > Thanks > > Were you trying to install scipy from source or using a package from a repository somewhere? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.djokovic at gmail.com Sun Aug 22 18:54:22 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Sun, 22 Aug 2010 18:54:22 -0400 Subject: [Numpy-discussion] numpy installation problems Message-ID: Actually I don't need scipy just need to get numpy working again. Scipy I downloaded a tar file from a mirror site. Sorry about the confusion-if I can get numpy working I would be happy. I have tried even installing an older version of numpy suitable for ferdora 6 Thanks a lot -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.djokovic at gmail.com Sun Aug 22 18:59:04 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Sun, 22 Aug 2010 18:59:04 -0400 Subject: [Numpy-discussion] numpy installation problems Message-ID: Do you think I should reinstall gcc? I do have the root password if nessasary Thanks for the replies -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Sun Aug 22 20:42:52 2010 From: david at silveregg.co.jp (David) Date: Mon, 23 Aug 2010 09:42:52 +0900 Subject: [Numpy-discussion] numpy installation problems In-Reply-To: References: Message-ID: <4C71C40C.8090002@silveregg.co.jp> On 08/23/2010 07:59 AM, martin djokovic wrote: > Do you think I should reinstall gcc? You most likely already have gcc, there is no need to reinstall it. The missing symbol refers to the math library from icc, so if you can find the missing libraries, you should be set: 1: get the name of the missing library by running ldd on the .so files which have missing symbols (fftpack_lite.so here). You should get a list of libraries, with some missing. 2: locate the missing libraries on your system. 3: if found, add the path of the libraries into LD_LIBRARY_PATH, e.g. export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH inside your shell 4: if not found, then you may need to reinstall numpy from sources. cheers, David From seb.haase at gmail.com Mon Aug 23 01:07:45 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 23 Aug 2010 07:07:45 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 9:41 AM, Sebastian Haase wrote: > On Sun, Aug 22, 2010 at 8:40 AM, Sebastian Haase wrote: >> On Sun, Aug 22, 2010 at 8:16 AM, Sebastian Haase wrote: >>> On Sun, Aug 22, 2010 at 3:39 AM, Sebastian Haase wrote: >>>> On Sun, Aug 22, 2010 at 12:02 AM, Christoph Gohlke wrote: >>>>> >>>>> >>>>> On 8/21/2010 2:37 PM, Sebastian Haase wrote: >>>>>> On Sat, Aug 21, 2010 at 11:29 PM, Christoph Gohlke ?wrote: >>>>>>> >>>>>>> >>>>>>> On 8/21/2010 1:44 PM, Sebastian Haase wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> this is somewhat OT for this list, but since I know that David and >>>>>>>> many others here have lot's of experience compiling C extensions I >>>>>>>> thought I could just ask: >>>>>>>> Looking at >>>>>>>> http://sourceforge.net/projects/mingw-w64/files/ >>>>>>>> I did not know (even after reading the FAQ) which file to download and >>>>>>>> how things would eventually work. >>>>>>>> >>>>>>>> I have a 64bit windows 7 installed, and got many precompiled packages >>>>>>>> for amd64 Python 2.7 from >>>>>>>> http://www.lfd.uci.edu/~gohlke/pythonlibs/ >>>>>>>> (thanks to ?Christoph Gohlke for all the work) >>>>>>>> But now I have some C++ extensions on my own, and know how build them >>>>>>>> using cygwin -- but that would only produce 32bit modules and should >>>>>>>> be unusable. >>>>>>>> >>>>>>>> So, the question is if someone has or knows of some tutorial about how >>>>>>>> to go about this - step by step. This info could maybe even go the >>>>>>>> scipy wiki.... >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sebastian Haase >>>>>>> >>>>>>> >>>>>>> Hi Sebastian, >>>>>>> >>>>>>> I am not aware of such a tutorial. There's some information at >>>>>>> >>>>>>> >>>>>>> I did not have good experience last time (about a year ago) I tried >>>>>>> mingw-w64. Occasional crashes during compilation and at runtime. >>>>>>> Probably that has changed. At least you have to create the missing >>>>>>> libpython and libmsvcr90 libraries from the dlls and make libmsvcr90 the >>>>>>> default crt. >>>>>>> >>>>>>> You probably know that the "free" Windows 7 Platform SDK can be used to >>>>>>> build Python>=2.6 extensions written in C89. >>>>>>> >>>>>>> >>>>>>> -- >>>>>> Hi Christoph, >>>>>> >>>>>> I did not exactly know this - thanks for the info (I knew about >>>>>> something called Visual Studio Express 2003- but that only >>>>>> works/worked for Python 2.5, I think...) >>>>> >>>>> You can use Visual Studio Express 2008 for building 32 bit extensions >>>>> for Python >=2.6. >>>>> >>>>>> >>>>>> Rephrasing my original question: Is the mingw-w64 at all "easy" by now >>>>> >>>>> Don't know. David Cournapeau probably has the most experience. >>>>> >>>>> http://bugs.python.org/issue4709 >>>>> http://www.equation.com/servlet/equation.cmd?call=fortran >>>>> >>>>>> ? How about cross compiling to 64bit Windows from a 32bit Ubuntu (that >>>>>> I could easily run on virtualbox) ? >>>>> >>>>> I am also interested in cross compiling on Ubuntu but have not found the >>>>> time to get started. The IOCBio project cross-compiles their 32 bit >>>>> extensions on Linux >>>>> . >>>>> But as you can see they use Wine and Mingw... >>>>> >>>>>> >>>>>> (But I'm not apposed at all to the "free" Windows 7 Platform SDK, so >>>>>> I'll look into that -- giant download !?) >>>>> >>>>> About one GB. >>>>> >>>>>> >>>> Do you know if that contains a C++ compiler ? ?The first page before >>>> it starts the actual download has "Visual C++ Compilers" grayed out >>>> ... !? >>>> >>>> -Sebastian >>>> >>> Ok, apparently I had to install the "dot NET Framework 4" from >>> http://msdn.microsoft.com/en-us/netframework/aa569263.aspx >>> first, before then the C++ could be installed. >>> But now setup.py still complains: >>> ? ? ? error: Unable to find vcvarsall.bat >>> and I think it is looking for >>> C:\Program Files (x86)\Microsoft Visual Studio 9.0 >>> while that file got installed in >>> C:\Program Files (x86)\Microsoft Visual Studio 10.0 >>> I don't know how to get the "log.debug" messages from the setup.py >>> script activated... >>> ? >> >> I got the debug out by adding a "-d" >> \Python27\python.exe setup.py -v build >> (even though setup.py --help ?states that --verbose should be already >> the default, well...) >> >> Then I mentioned the environment variable %VS90COMNTOOLS% >> which I then set to >> $$>set VS90COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin >> >> But now I'm getting a traceback: >> ?File "C:\Python27\lib\distutils\msvc9compiler.py", line 469, in compile >> ? ?self.initialize() >> ?File "C:\Python27\lib\distutils\msvc9compiler.py", line 379, in initialize >> ? ?vc_env = query_vcvarsall(VERSION, plat_spec) >> ?File "C:\Python27\lib\distutils\msvc9compiler.py", line 295, in >> query_vcvarsall >> ? ?raise ValueError(str(list(result.keys()))) >> ValueError: [u'path'] >> >> Well, still trying .... >> Cheers, >> Sebastian >> > Now, I renamed the file > C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat > (as mentioned in > http://mattptr.net/2010/07/28/building-python-extensions-in-a-modern-windows-environment/) > to > vcvars64.bat > and moved it to > C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64 > > The compiler races through and I end up getting linker problems: > LINK : error LNK2001: unresolved external symbol init_sebmodule > So this last linker error was fixed by changing my setup.py to use _seb instead of _sebmodule as external extension. All seems to work now -- with the .NET 4 and VS 10. If it was binding to the wrong CRT version, I cannot see any problem yet. (I did set DISTUTILS_USE_SDK and MSSdk either, I mostly just followed the referenced site http://mattptr... and did the modification I described) My extensions implement simple numeric routines and some OpenGL code. Thanks, Sebastian From stefan at sun.ac.za Mon Aug 23 04:28:45 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 23 Aug 2010 10:28:45 +0200 Subject: [Numpy-discussion] Using numpy's flatten_dtype with structured dtypes that have titles In-Reply-To: References: Message-ID: Hi Emma On 19 August 2010 23:07, Emma Willemsma wrote: > I am working with structured arrays to store experimental data. I'm using > titles to store information about my fields, in this case the units of > measure. When I call numpy.lib.io.flatten_dtype() on my dtype, I get: > > ValueError: too many values to unpack > File "c:\Python25\Lib\site-packages\numpy\lib\_iotools.py", line 78, in > flatten_dtype Could you please provide an example of your dtype, or of your .txt file? With such a test case, we should be able to make flatten_dtype more robust. > The user who responded suggested the following monkey-patch: Unfortunately, the monkey patch is aimed at addressing your issue only, so we won't be able to use it as is. Regards St?fan From martin.djokovic at gmail.com Mon Aug 23 10:17:49 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Mon, 23 Aug 2010 10:17:49 -0400 Subject: [Numpy-discussion] reply: numpy installation problems Message-ID: Hello David, Thanks for your kind reply-I did do ldd and got the following [xyz at papageno fft]$ ldd fftpack_lite.so linux-gate.so.1 => (0x0060d000) libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x0025e000) libm.so.6 => /lib/libm.so.6 (0x00110000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00bad000) libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x00ecc000) libc.so.6 => /lib/libc.so.6 (0x0049c000) libdl.so.2 => /lib/libdl.so.2 (0x008fd000) /lib/ld-linux.so.2 (0x00243000) So there are no missing libraries-but while trying to figure out how to use the LD_LIBRARY_PATH I read (internet) that its not good to use -so I was wondering if I installed NUMPY from source, will it solve the problem? When you say "from source" does it mean the tar file? I already did that python setup.py install Should I use some flags? Didn't say so in my README file. I am sorry for sounding so dumb but this is the first time I am trying to install something like this on a LINUX system. Thank you very much -------------- next part -------------- An HTML attachment was scrubbed... URL: From emma.willemsma at morgansolar.com Mon Aug 23 10:50:59 2010 From: emma.willemsma at morgansolar.com (Emma Willemsma) Date: Mon, 23 Aug 2010 10:50:59 -0400 Subject: [Numpy-discussion] Using numpy's flatten_dtype with structured dtypes that have titles In-Reply-To: References: Message-ID: I have attached a short script and sample text file that demonstrate the problem. The dtype I'm using in the example is: dtype2 = numpy.dtype([(("Amps","Current"),"f8"),(("Volts","Voltage"),"f8"),(("Watts","Power"),"f8")]) When I run the code as-is, I get the ValueError. If I run it with the monkey patch, it runs as I would expect. On the other hand, if I remove the titles from the dtype, the code works with or without the monkey patch. Without the titles, the dtype looks like: dtype1 = numpy.dtype([("Current","f8"),("Voltage","f8"),("Power","f8")]) Thanks very much for your help with this issue. Emma 2010/8/23 St?fan van der Walt > Hi Emma > > On 19 August 2010 23:07, Emma Willemsma > wrote: > > I am working with structured arrays to store experimental data. I'm using > > titles to store information about my fields, in this case the units of > > measure. When I call numpy.lib.io.flatten_dtype() on my dtype, I get: > > > > ValueError: too many values to unpack > > File "c:\Python25\Lib\site-packages\numpy\lib\_iotools.py", line 78, in > > flatten_dtype > > Could you please provide an example of your dtype, or of your .txt > file? With such a test case, we should be able to make flatten_dtype > more robust. > > > The user who responded suggested the following monkey-patch: > > Unfortunately, the monkey patch is aimed at addressing your issue > only, so we won't be able to use it as is. > > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- 1.000 -0.000 -0.000 0.990 0.356 0.352 0.980 0.409 0.401 0.970 0.440 0.427 0.960 0.462 0.444 0.950 0.479 0.455 0.940 0.493 0.464 0.930 0.505 0.470 0.920 0.515 0.474 0.910 0.524 0.477 0.900 0.533 0.479 0.890 0.540 0.480 0.880 0.547 0.481 0.870 0.553 0.481 0.860 0.558 0.480 0.850 0.564 0.479 0.840 0.569 0.478 0.830 0.573 0.476 0.820 0.578 0.474 0.810 0.582 0.471 0.800 0.586 0.469 0.790 0.590 0.466 0.780 0.593 0.463 0.770 0.597 0.459 0.760 0.600 0.456 0.750 0.603 0.452 0.740 0.606 0.449 0.730 0.609 0.445 0.720 0.612 0.441 0.710 0.615 0.436 0.700 0.617 0.432 0.690 0.620 0.428 0.680 0.622 0.423 0.670 0.624 0.418 0.660 0.627 0.414 0.650 0.629 0.409 0.640 0.631 0.404 0.630 0.633 0.399 0.620 0.635 0.394 0.610 0.637 0.389 0.600 0.639 0.384 0.590 0.641 0.378 0.580 0.643 0.373 0.570 0.645 0.368 0.560 0.647 0.362 0.550 0.648 0.357 0.540 0.650 0.351 0.530 0.652 0.345 0.520 0.653 0.340 0.510 0.655 0.334 0.500 0.656 0.328 0.490 0.658 0.322 0.480 0.660 0.317 0.470 0.661 0.311 0.460 0.662 0.305 0.450 0.664 0.299 0.440 0.665 0.293 0.430 0.667 0.287 0.420 0.668 0.281 0.410 0.669 0.274 0.400 0.671 0.268 0.390 0.672 0.262 0.380 0.673 0.256 0.370 0.674 0.249 0.360 0.676 0.243 0.350 0.677 0.237 0.340 0.678 0.230 0.330 0.679 0.224 0.320 0.680 0.218 0.310 0.681 0.211 0.300 0.682 0.205 0.290 0.684 0.198 0.280 0.685 0.192 0.270 0.686 0.185 0.260 0.687 0.179 0.250 0.688 0.172 0.240 0.689 0.165 0.230 0.690 0.159 0.220 0.691 0.152 0.210 0.692 0.145 0.200 0.693 0.139 0.190 0.694 0.132 0.180 0.695 0.125 0.170 0.696 0.118 0.160 0.696 0.111 0.150 0.697 0.105 0.140 0.698 0.098 0.130 0.699 0.091 0.120 0.700 0.084 0.110 0.701 0.077 0.100 0.702 0.070 0.090 0.703 0.063 0.080 0.703 0.056 0.070 0.704 0.049 0.060 0.705 0.042 0.050 0.706 0.035 0.040 0.707 0.028 0.030 0.708 0.021 0.020 0.708 0.014 0.010 0.709 0.007 1.000 0.000 0.000 1.000 0.007 0.007 1.000 0.014 0.014 0.999 0.021 0.021 0.999 0.028 0.028 0.999 0.035 0.035 0.999 0.043 0.043 0.999 0.050 0.050 0.998 0.057 0.057 0.998 0.064 0.064 0.998 0.071 0.071 0.998 0.078 0.078 0.998 0.085 0.085 0.997 0.092 0.092 0.997 0.099 0.099 0.997 0.106 0.106 0.997 0.114 0.113 0.997 0.121 0.120 0.996 0.128 0.127 0.996 0.135 0.134 0.996 0.142 0.141 0.996 0.149 0.148 0.996 0.156 0.155 0.995 0.163 0.163 0.995 0.170 0.170 0.995 0.177 0.177 0.995 0.185 0.184 0.995 0.192 0.191 0.994 0.199 0.198 0.994 0.206 0.205 0.994 0.213 0.212 0.994 0.220 0.219 0.994 0.227 0.226 0.993 0.234 0.233 0.993 0.241 0.240 0.993 0.248 0.247 0.993 0.256 0.254 0.993 0.263 0.261 0.992 0.270 0.268 0.992 0.277 0.275 0.992 0.284 0.282 0.992 0.291 0.289 0.992 0.298 0.296 0.991 0.305 0.303 0.991 0.312 0.310 0.991 0.319 0.317 0.991 0.327 0.324 0.991 0.334 0.331 0.990 0.341 0.337 0.990 0.348 0.344 0.990 0.355 0.351 0.989 0.362 0.358 0.987 0.369 0.365 0.986 0.376 0.371 0.985 0.383 0.378 0.983 0.390 0.384 0.982 0.398 0.390 0.981 0.405 0.397 0.979 0.412 0.403 0.977 0.419 0.409 0.974 0.426 0.415 0.972 0.433 0.421 0.970 0.440 0.427 0.967 0.447 0.432 0.963 0.454 0.438 0.960 0.461 0.443 0.956 0.469 0.448 0.952 0.476 0.453 0.947 0.483 0.457 0.942 0.490 0.462 0.937 0.497 0.466 0.931 0.504 0.469 0.924 0.511 0.472 0.917 0.518 0.475 0.909 0.525 0.477 0.900 0.532 0.479 0.890 0.540 0.480 0.880 0.547 0.481 0.868 0.554 0.481 0.855 0.561 0.480 0.842 0.568 0.478 0.826 0.575 0.475 0.810 0.582 0.471 0.791 0.589 0.466 0.771 0.596 0.460 0.749 0.603 0.452 0.725 0.611 0.442 0.698 0.618 0.431 0.669 0.625 0.418 0.637 0.632 0.402 0.602 0.639 0.385 0.563 0.646 0.364 0.521 0.653 0.341 0.475 0.660 0.314 0.425 0.667 0.283 0.369 0.674 0.249 0.308 0.682 0.210 0.241 0.689 0.166 0.168 0.696 0.117 0.088 0.703 0.062 -------------- next part -------------- A non-text attachment was scrubbed... Name: test5.py Type: application/octet-stream Size: 1060 bytes Desc: not available URL: From pgmdevlist at gmail.com Mon Aug 23 11:09:17 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 23 Aug 2010 11:09:17 -0400 Subject: [Numpy-discussion] Using numpy's flatten_dtype with structured dtypes that have titles In-Reply-To: References: Message-ID: <8C87EC92-0BCB-436F-9683-9271BB80B20A@gmail.com> On Aug 23, 2010, at 10:50 AM, Emma Willemsma wrote: > I have attached a short script and sample text file that demonstrate the problem. The dtype I'm using in the example is: > > dtype2 = numpy.dtype([(("Amps","Current"),"f8"),(("Volts","Voltage"),"f8"),(("Watts","Power"),"f8")]) > > When I run the code as-is, I get the ValueError. If I run it with the monkey patch, it runs as I would expect. > > On the other hand, if I remove the titles from the dtype, the code works with or without the monkey patch. Without the titles, the dtype looks like: > > dtype1 = numpy.dtype([("Current","f8"),("Voltage","f8"),("Power","f8")]) > > Thanks very much for your help with this issue. > > Emma Emma, Please open a ticket. That's a bug in flatten_dtype, which cannot handle titles in the dtype (as I never used titles, I forgot they were there...). I'll try to take care of that ASAP. From ben.root at ou.edu Mon Aug 23 11:44:03 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 23 Aug 2010 10:44:03 -0500 Subject: [Numpy-discussion] reply: numpy installation problems In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 9:17 AM, martin djokovic wrote: > Hello David, > > Thanks for your kind reply-I did do ldd and got the following > > [xyz at papageno fft]$ ldd fftpack_lite.so > > linux-gate.so.1 => (0x0060d000) > > libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x0025e000) > > libm.so.6 => /lib/libm.so.6 (0x00110000) > > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00bad000) > > libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x00ecc000) > > libc.so.6 => /lib/libc.so.6 (0x0049c000) > > libdl.so.2 => /lib/libdl.so.2 (0x008fd000) > > /lib/ld-linux.so.2 (0x00243000) > > > So there are no missing libraries-but while trying to figure out how to use > the LD_LIBRARY_PATH > > I read (internet) that its not good to use -so I was wondering if I > installed NUMPY from source, will it solve the problem? > > When you say "from source" does it mean the tar file? I already did that > > python setup.py install > > Should I use some flags? Didn't say so in my README file. I am sorry for > sounding so dumb but this is the first time I am trying to install something > like this on a LINUX system. > > > Thank you very much > > martin, try: sudo python setup.py install It will ask for your root password. If that fails, then you are probably not on the sudo-ers list. Therefore, you should log in as root (in the terminal only) by doing: su - This will put you in the /root directory. You will have to navigate back to where you were building numpy and run the 'python setup.py install' command as root. If that doesn't work, delete the 'build' directory and try again. When done being root, then you can type 'exit' and you will be back as a regular user. I hope that helps, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From emma.willemsma at morgansolar.com Mon Aug 23 12:10:29 2010 From: emma.willemsma at morgansolar.com (Emma Willemsma) Date: Mon, 23 Aug 2010 12:10:29 -0400 Subject: [Numpy-discussion] Using numpy's flatten_dtype with structured dtypes that have titles In-Reply-To: <8C87EC92-0BCB-436F-9683-9271BB80B20A@gmail.com> References: <8C87EC92-0BCB-436F-9683-9271BB80B20A@gmail.com> Message-ID: Thank you Pierre, I have opened the ticket as requested. On Mon, Aug 23, 2010 at 11:09 AM, Pierre GM wrote: > > On Aug 23, 2010, at 10:50 AM, Emma Willemsma wrote: > > > I have attached a short script and sample text file that demonstrate the > problem. The dtype I'm using in the example is: > > > > dtype2 = > numpy.dtype([(("Amps","Current"),"f8"),(("Volts","Voltage"),"f8"),(("Watts","Power"),"f8")]) > > > > When I run the code as-is, I get the ValueError. If I run it with the > monkey patch, it runs as I would expect. > > > > On the other hand, if I remove the titles from the dtype, the code works > with or without the monkey patch. Without the titles, the dtype looks like: > > > > dtype1 = numpy.dtype([("Current","f8"),("Voltage","f8"),("Power","f8")]) > > > > Thanks very much for your help with this issue. > > > > Emma > > Emma, > Please open a ticket. That's a bug in flatten_dtype, which cannot handle > titles in the dtype (as I never used titles, I forgot they were there...). > I'll try to take care of that ASAP. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Aug 23 12:20:47 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 23 Aug 2010 11:20:47 -0500 Subject: [Numpy-discussion] questions about a "complicated" user-defined dtype and the ufunc API In-Reply-To: References: Message-ID: On Aug 22, 2010, at 4:36 PM, Nathaniel Smith wrote: > I'm experimenting with a user-defined "enumeration" dtype -- where the > underlying array holds a set of integers, but they (mostly) appear to > the user as strings. (This would be potentially useful for > representing categorical data, modeling hdf5 enumerations, etc.) So > for each set of enumerated values, I have one Python-level object that > stores the mapping between strings and integers (an instance of the > 'Enum' class), and a few Python-level objects that represent each of > the enumerated values (instances of the 'EnumValue' class). > > To map this into numpy, I've defined and registered a single custom > dtype with 'EnumValue' as its typeobj, and then when I want to create > an array of enumerations I make a copy of this registered dtype (with > PyArray_DescrNewFromType) and stash a reference to the appropriate > Enum object in its 'metadata' dictionary. > > Question 1: Is this overall approach -- of only calling > PyArray_RegisterDataType once, and then sharing the resulting typenum > among all my different dtype instances -- correct? It doesn't seem > reasonable to register a new dtype for every different set of > enumerations, because AFAICT this would create a memory leak (since > you can't "unregister" a dtype). Yes, this is the approach to take. > > Anyway, that seems to be working okay, but then I wanted to teach "==" > about my new dtype, so that I can compare to strings and such. So, I > need to register my comparison function with the "np.equal" ufunc. > > Question 2: Am I missing something, or does the ufunc API make this > impossible? The problem is that a "PyUFuncGenericFunction" doesn't > have any way to find the dtypes of the arrays that it's working on. > All of the PyArray_ArrFuncs functions take a pointer to the underlying > ndarray as an argument, so that when working with a string or void > array, you can find the actual dtype and figure out (e.g.) the size of > the objects involved. But ufunc inner loops don't get that, so I guess > it's just impossible to define a ufunc over variable-sized data, or > data that you need to be able to see the dtype metadata to interpret? Yes, currently that is correct. Variable data-types don't work in ufuncs for some subtle reasons. But, the changes that allow date-times to work also fix (or will fix) this problem as a side-effect. The necessary changes to ufuncs have not been made, yet, however. They are planned. And, yes, this would allow ufuncs to be used for string equality testing, etc. Thanks, -Travis From opossumnano at gmail.com Mon Aug 23 12:50:09 2010 From: opossumnano at gmail.com (Tiziano Zito) Date: Mon, 23 Aug 2010 18:50:09 +0200 Subject: [Numpy-discussion] bug in dtype.__eq__ method ? Message-ID: <20100823165009.GA5818@tulpenbaum.cognition.tu-berlin.de> hi all, we just noticed the following weird thing: $ python Python 2.6.6rc2 (r266rc2:84114, Aug 18 2010, 07:33:44) [GCC 4.4.5 20100816 (prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '2.0.0.dev8469' >>> numpy.dtype('float64') == None True >>> numpy.dtype('float32') == None False is this a bug in the dtype.__eq__ method? ciao, tiziano From zbyszek at in.waw.pl Mon Aug 23 12:55:26 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Mon, 23 Aug 2010 18:55:26 +0200 Subject: [Numpy-discussion] bug in dtype.__eq__ method ? In-Reply-To: <20100823165009.GA5818@tulpenbaum.cognition.tu-berlin.de> References: <20100823165009.GA5818@tulpenbaum.cognition.tu-berlin.de> Message-ID: <20100823165526.GB23997@in.waw.pl> On Mon, Aug 23, 2010 at 06:50:09PM +0200, Tiziano Zito wrote: > hi all, > we just noticed the following weird thing: > > $ python > Python 2.6.6rc2 (r266rc2:84114, Aug 18 2010, 07:33:44) > [GCC 4.4.5 20100816 (prerelease)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.version.version > '2.0.0.dev8469' Also with numpy.version.version == 1.3.0. Best, Zbyszek From martin.djokovic at gmail.com Mon Aug 23 13:06:34 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Mon, 23 Aug 2010 13:06:34 -0400 Subject: [Numpy-discussion] Re reply: numpy installation problems Message-ID: Hi Ben, Thanks but thats not working-I am already root-I have tried to give in detail what I am doing Used steps............................................ I downloaded numpy-all of the following and none worked-also I have a FEDORA CORE 6 with Python 2.4 in my machine. numpy-1.5.0b2.tar.gz numpy-1.4.1b2.tar.gz numpy-1.3.0b2.tar.gz The way the system is setup means that as root I cannot see my own Desktop (user Desktop) so I download the TAR file then move it to /scratch/ (a common area that the root and user can see) Then I do su - login as root then navigate to to /scratch/numpy/ Then do python setup.py install While installing (the above command) this is flaged in red-should I worry??? gnu: no Fortran 90 compiler found Then I do the following python (this works) >>>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 144, in ? import fft File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? from fftpack import * File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? import fftpack_lite as fftpack ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 I am really sorry if this is confusing. Thank you so much for your patiance in this matter -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Aug 23 14:16:15 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 23 Aug 2010 13:16:15 -0500 Subject: [Numpy-discussion] Re reply: numpy installation problems In-Reply-To: References: Message-ID: <4C72BAEF.60708@gmail.com> On 08/23/2010 12:06 PM, martin djokovic wrote: > Hi Ben, > > Thanks but thats not working-I am already root-I have tried to give in > detail what I am doing > > Used steps............................................ > > > I downloaded numpy-all of the following and none worked-also I have a > FEDORA CORE 6 with Python 2.4 in my machine. > > > numpy-1.5.0b2.tar.gz > > > numpy-1.4.1b2.tar.gz > > > numpy-1.3.0b2.tar.gz > > > > > The way the system is setup means that as root I cannot see my own > Desktop (user Desktop) so I download the TAR file then move it to > /scratch/ (a common area that the root and user can see) > > > Then I do > > su - > > login as root then navigate to to /scratch/numpy/ > > > Then do > > python setup.py install > > > > While installing (the above command) this is flaged in red-should I > worry??? > > gnu: no Fortran 90 compiler found > > > > > Then I do the following > > python (this works) > > >>>> import numpy > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line > 144, in ? > > import fft > > File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", > line 4, in ? > > from fftpack import * > > File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", > line 28, in ? > > import fftpack_lite as fftpack > > ImportError: > /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: > undefined symbol: vmldCos2 > > > > I am really sorry if this is confusing. > > Thank you so much for your patiance in this matter > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Given that the error pertains to the /usr/local directory, you may have python installed in more than one location. So please check the path to python is being used as a normal user and as root - such as doing the 'which python' command. Really you must remove the '/usr/local/lib/python2.4/site-packages/numpy/' directory and other numpy files *before* installing numpy and scipy. Then you probably need to specific the full path to the python you want to use - something like: $/usr/local/bin/python setup.py install While I doubt that your current setup is controlled by your Fedora 6 package manager, you should be able to determine if numpy was originally installed via the distro by querying the installed packages. Your package manager should show it as installed if you search for numpy or just use rpm: $rpm -qa | numpy If so, then you probably can do: $yum reinstall numpy You may be able to yum or whatever package manager to install both numpy and scipy if these are both in Fedora 6 repositiories (they are for Fedora 13). For example, as root: $ yum install numpy scipy Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.djokovic at gmail.com Mon Aug 23 15:29:45 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Mon, 23 Aug 2010 15:29:45 -0400 Subject: [Numpy-discussion] still installation problems for numpy Message-ID: I tried to find all the numpy locations and removed them then tried.... yum install numpy After trying for s few minutes.......got the following Error: Cannot find a valid baseurl for repo: extras If I install the other way using the tar file ...I still get >>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 144, in ? import fft File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? from fftpack import * File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? import fftpack_lite as fftpack ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 I tried looking for missing libraries using... ldd fft/fftpack_lite.so No libraries missing...... linux-gate.so.1 => (0x0060d000) libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x0025e000) libm.so.6 => /lib/libm.so.6 (0x00110000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00bad000) libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x00ecc000) libc.so.6 => /lib/libc.so.6 (0x0049c000) libdl.so.2 => /lib/libdl.so.2 (0x008fd000) /lib/ld-linux.so.2 (0x00243000) checked the path to python for both root and user-made an alias in both to refer to same python Nothing seems to work-if you guys have any more ideas I would love to hear them-I am stuck here amd cannot move ahead and no one in my office who can help-so your help is much appreciated. I am also wondering if you need to scipy or biopython to use numpy??? Thanj=ks -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.djokovic at gmail.com Mon Aug 23 15:37:54 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Mon, 23 Aug 2010 15:37:54 -0400 Subject: [Numpy-discussion] still having numpy installation issues Message-ID: I tried to find all the numpy locations and removed them then tried.... yum install numpy After trying for s few minutes.......got the following Error: Cannot find a valid baseurl for repo: extras If I install the other way using the tar file ...I still get >>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 144, in ? import fft File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? from fftpack import * File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? import fftpack_lite as fftpack ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 I tried looking for missing libraries using... ldd fft/fftpack_lite.so No libraries missing...... linux-gate.so.1 => (0x0060d000) libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x0025e000) libm.so.6 => /lib/libm.so.6 (0x00110000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00bad000) libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x00ecc000) libc.so.6 => /lib/libc.so.6 (0x0049c000) libdl.so.2 => /lib/libdl.so.2 (0x008fd000) /lib/ld-linux.so.2 (0x00243000) checked the path to python for both root and user-made an alias in both to refer to same python Nothing seems to work-if you guys have any more ideas I would love to hear them-I am stuck here amd cannot move ahead and no one in my office who can help-so your help is much appreciated. I am also wondering if you need to scipy or biopython to use numpy??? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Mon Aug 23 15:59:57 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 23 Aug 2010 14:59:57 -0500 Subject: [Numpy-discussion] still having numpy installation issues In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 2:37 PM, martin djokovic wrote: > I tried to find all the numpy locations and removed them then tried.... > > > yum install numpy > > > After trying for s few minutes.......got the following > > > Error: Cannot find a valid baseurl for repo: extras > > > > If I install the other way using the tar file ...I still get > > > >>> import numpy > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 144, in ? > import fft > > > File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? > from fftpack import * > File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? > > > import fftpack_lite as fftpack > ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 > > > I tried looking for missing libraries using... > > ldd fft/fftpack_lite.so > > No libraries missing...... > > linux-gate.so.1 => (0x0060d000) > > libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x0025e000) > > libm.so.6 => /lib/libm.so.6 (0x00110000) > > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00bad000) > > > libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x00ecc000) > > libc.so.6 => /lib/libc.so.6 (0x0049c000) > > libdl.so.2 => /lib/libdl.so.2 (0x008fd000) > > /lib/ld-linux.so.2 (0x00243000) > > checked the path to python for both root and user-made an alias in both to > refer to same python > > > Nothing seems to work-if you guys have any more ideas I would love to > hear them-I am stuck here amd cannot move ahead and no one in my office who > can help-so your help is much appreciated. > > > I am also wondering if you need to scipy or biopython to use numpy??? > Martin, Fedora 6 has been out-of-date for a while and the repositories are no longer available. You might get lucky with 'yum reinstall numpy', but 'yum install numpy' would not work because the repos are not around anymore. If the package approach does not work (which is the preferred method), then you need to install from the source (tar file). Be sure to remove the build directory first before running 'python setup.py install'. This should definitely work because you have cleaned up any possible conflicting versions of numpy. There is still a chance that your system is so out-of-date that the build will fail. As a last note, numpy does not require scipy (actually, scipy requires numpy...). Ben Root P.S. - For archive and searching purposes, please keep the discussion on a single thread. The best way to do that is to "Reply to all" to a response from us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Aug 23 16:00:43 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 23 Aug 2010 15:00:43 -0500 Subject: [Numpy-discussion] still having numpy installation issues In-Reply-To: References: Message-ID: <4C72D36B.4080008@gmail.com> On 08/23/2010 02:37 PM, martin djokovic wrote: > > I tried to find all the numpy locations and removed them then tried.... > > > yum install numpy > > > After trying for s few minutes.......got the following > > > Error: Cannot find a valid baseurl for repo: extras > > > > If I install the other way using the tar file ...I still get > > > >>> import numpy > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 144, in ? > import fft > > File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? > from fftpack import * > File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? > > import fftpack_lite as fftpack > ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 > > > I tried looking for missing libraries using... > > ldd fft/fftpack_lite.so > > No libraries missing...... > > linux-gate.so.1 => (0x0060d000) > > libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x0025e000) > > libm.so.6 => /lib/libm.so.6 (0x00110000) > > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00bad000) > > > libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x00ecc000) > > libc.so.6 => /lib/libc.so.6 (0x0049c000) > > libdl.so.2 => /lib/libdl.so.2 (0x008fd000) > > /lib/ld-linux.so.2 (0x00243000) > > checked the path to python for both root and user-made an alias in > both to refer to same python > > > Nothing seems to work-if you guys have any more ideas I would love to > hear them-I am stuck here amd cannot move ahead and no one in my > office who can help-so your help is much appreciated. > > > > I am also wondering if you need to scipy or biopython to use numpy??? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion The issue is that you have not removed all numpy locations because you are not looking in the correct places. So you really must find which python is being used as a normal user and as root such as using the 'which python' command as both a normal user and as root. You clearly have a numpy here '/usr/local/lib/python2.4/site-packages/numpy' You need to remove that directory (and all contents of it) as root (because of the permissions). But you need to know which python is being used before reinstalling and use that one for the installation. You need numpy for both scipy and biopython so those will not help you. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedrichromstedt at gmail.com Mon Aug 23 16:04:55 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Mon, 23 Aug 2010 22:04:55 +0200 Subject: [Numpy-discussion] numpy installation problems In-Reply-To: References: Message-ID: 2010/8/23 martin djokovic : > /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined > symbol: vmldCos2 To me this looks familiar ... I ran into this problem usually when having Python compiled with another compiler than the library. In your case, it's a bit strange, because Python distutils, for compiling numpy, picks up the compiler used for compiling Python. But, obviously, the numpy .so lib loads, just the dependency not. So I think the proposition that you have /two/ Pythons, one which you run as user, and one which you run when installing numpy, looks not like a promising explanation. Can you anyway do the folling: (as normal user)$ which python (as user installing numpy)# which python ? Also you can try if the "import numpy" runs well when logged in as the user installing numpy. But I believe it doesn't About the Fortran compiler, don't worry. Can you post some snippet of the build process, to see what compiler is issued? (I'm curious because: 1. If it issues an older "gcc" than that used to compile the .so lib used by the numpy .so lib, 2. this would explain the load error.) Use "python setup.py build | tee log-build.txt 2>&1" to get the full log of the build in a log file "log-build.txt". Let the 2>&1 alone if you're not on bash, but then the error msg will not be logged probably. Don't be afraid, "setup.py build", as the name says, installs nothing, to install one uses "setup.py install" (which in turn triggers a build). hth, Friedrich From oliphant at enthought.com Mon Aug 23 16:30:19 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 23 Aug 2010 15:30:19 -0500 Subject: [Numpy-discussion] Github migration? Message-ID: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Hi all, I'm curious as to the status of the Github migration and if there is anything I can do to help. I have a couple of weeks right now and I would love to see us make the transition of both NumPy and SciPy to GIT. On a slightly related note, it would really help the numpy-refactor project if it received more input from others in the community. Right now, the numpy-refactor is happening on a public github branch (named numpy-refactor) awaiting numpy itself to be on github. It would be more useful if the numpy-refactor branch were a regular branch of the github NumPy project. The numpy-refactor project is making great progress and we have a working core library that can be built on Windows, Mac, and Linux. The goal is for this numpy-refactor to become the basis for NumPy 2.0 which should come out sometime this Fall. Already, a lot of unit tests have been written and code coverage has increased on the core NumPy code which I think we all agree is "a good thing" In addition, some lingering reference count bugs (particularly in object arrays) have been found and squashed. There is also some good progress on the Cython backend for .NET which would allow and also put pressure on migration of most of SciPy to Cython-or-Fwrap generated extension modules. I am looking forward to working on NumPy a little more over the coming months. All the best, -Travis From martin.djokovic at gmail.com Mon Aug 23 16:57:36 2010 From: martin.djokovic at gmail.com (martin djokovic) Date: Mon, 23 Aug 2010 16:57:36 -0400 Subject: [Numpy-discussion] further to my previous numpy installation problems Message-ID: First of all sorry about not posting as a reply in the above thread-for some reason it does not seem to work So have to post as a new question-sorry about that. Ok so got a tar file installed as root, then did LOCATE NUMPY All most all the files found with numpy were in /usr/local/lib/python2.4/site-packages/numpy Which is where I installed numpy......... Both root and user are set to use python 2.4 However the following were also listed- I as root can remove them but I am scared that that might mess something else-should I remove them and try again? I am specially concerned about the redhat/SOURCES and also the first entry (which I did remove before installing this time but has come back) /usr/lib/python2.4/site-packages/numpy /usr/local/share/pymol/ext/lib/python2.4/site-packages/Numeric/_numpy.so /usr/share/doc/numpy-1.0.3 /usr/src/redhat/SOURCES/numpy-1.0.1-f2py.patch /usr/src/redhat/SOURCES/numpy-1.0.3-2.tar.gz /usr/src/redhat/SOURCES/numpy-1.0.3-setup.patch /usr/src/redhat/SPECS/numpy.spec /scratch/numpy-1.0.4 /usr/bin/f2py.numpy /usr/lib/python2.4/site-packages/Numeric/_numpy.so /usr/local/biopython-1.53/numpy /usr/local/lib/python2.4/site-packages/Numeric/_numpy.so -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Aug 23 17:15:19 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 23 Aug 2010 23:15:19 +0200 Subject: [Numpy-discussion] Github migration? In-Reply-To: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On 23 August 2010 22:30, Travis Oliphant wrote: > I'm curious as to the status of the Github migration and if there is anything I can do to help. ?I have a couple of weeks right now and I would love to see us make the transition of both NumPy and SciPy to GIT. The test repo has been on github.com/numpy for a while now, without any problems reported (I think at least Pauli tried it!). The last I heard, there were no objections, and release-manager Ralf was also on board. Shall we hold our breath and make the final leap? Regards St?fan From pav at iki.fi Mon Aug 23 17:15:55 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 23 Aug 2010 21:15:55 +0000 (UTC) Subject: [Numpy-discussion] Github migration? References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Mon, 23 Aug 2010 15:30:19 -0500, Travis Oliphant wrote: > I'm curious as to the status of the Github migration and if there is > anything I can do to help. I have a couple of weeks right now and I > would love to see us make the transition of both NumPy and SciPy to GIT. I think the more or less latest state is here: http://github.com/pv/numpy2git with the svn-all-fast-export branch probably being the way to go. The thing that is missing is comparison of each Git revision in the converted repo to the corresponding revision in SVN. We ran into some bugs in svn-all-fast-export which caused some commits in the history to have the wrong content -- so to be sure, we have to do a brute-force comparison of the tree against SVN for each commit. The particular bug here was fixed in the conversion tool, but better safe than sorry. This shouldn't be too difficult to script, but hasn't been done yet. It should be done immediately after the "export" phase in the Makefile -- i.e., immediately after svn-all-fast-export. An SVN dump for testing can be found here: http://pav.iki.fi/tmp/numpy.svndump.gz > On a slightly related note, it would really help the numpy-refactor > project if it received more input from others in the community. Right > now, the numpy-refactor is happening on a public github branch (named > numpy-refactor) awaiting numpy itself to be on github. It would be > more useful if the numpy-refactor branch were a regular branch of the > github NumPy project. I'll try to find some time next-week-ish to look at this... Pauli From jsseabold at gmail.com Mon Aug 23 17:17:00 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 23 Aug 2010 17:17:00 -0400 Subject: [Numpy-discussion] Datarray sprint July 28, 2010 In-Reply-To: References: Message-ID: On Thu, Jul 29, 2010 at 1:06 PM, Keith Goodman wrote: > Going forward we would like to: > > - Move the repo from Fernando's personal github account to a datarray > account. But we do not know a way to move a github repo along with all > its meta data (fork queue, issue tracker etc). Is Fernando's github still the most up to date location for datarray? Skipper From kwgoodman at gmail.com Mon Aug 23 17:19:48 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 23 Aug 2010 14:19:48 -0700 Subject: [Numpy-discussion] Datarray sprint July 28, 2010 In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 2:17 PM, Skipper Seabold wrote: > Is Fernando's github still the most up to date location for datarray? Yes. From stefan at sun.ac.za Mon Aug 23 17:26:25 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 23 Aug 2010 23:26:25 +0200 Subject: [Numpy-discussion] Using numpy's flatten_dtype with structured dtypes that have titles In-Reply-To: <8C87EC92-0BCB-436F-9683-9271BB80B20A@gmail.com> References: <8C87EC92-0BCB-436F-9683-9271BB80B20A@gmail.com> Message-ID: On 23 August 2010 17:09, Pierre GM wrote: >> On the other hand, if I remove the titles from the dtype, the code works with or without the monkey patch. Without the titles, the dtype looks like: >> >> dtype1 = numpy.dtype([("Current","f8"),("Voltage","f8"),("Power","f8")]) >> >> Thanks very much for your help with this issue. >> >> Emma > > Emma, > Please open a ticket. That's a bug in flatten_dtype, which cannot handle titles in the dtype (as I never used titles, I forgot they were there...). I'll try to take care of that ASAP. Not sure, but the fix may be as simple as diff --git a/numpy/lib/_iotools.py b/numpy/lib/_iotools.py index c9d8104..3a2a62b 100644 --- a/numpy/lib/_iotools.py +++ b/numpy/lib/_iotools.py @@ -132,7 +132,8 @@ def flatten_dtype(ndtype, flatten_base=False): else: types = [] for field in names: - (typ, _) = ndtype.fields[field] + fieldinfo = ndtype.fields[field] + typ = fieldinfo[0] flat_dt = flatten_dtype(typ, flatten_base) types.extend(flat_dt) return types Regards St?fan From stefan at sun.ac.za Mon Aug 23 17:31:14 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 23 Aug 2010 23:31:14 +0200 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On 23 August 2010 23:15, Pauli Virtanen wrote: > The thing that is missing is comparison of each Git revision in the > converted repo to the corresponding revision in SVN. > > We ran into some bugs in svn-all-fast-export which caused some commits in > the history to have the wrong content -- so to be sure, we have to do a > brute-force comparison of the tree against SVN for each commit. The > particular bug here was fixed in the conversion tool, but better safe > than sorry. Erk. What's the quickest route to go: compare the actual patches, or bring a tree up to date for each revision and compute some sort of working-copy checksum? Regards St?fan From pav at iki.fi Mon Aug 23 17:36:22 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 23 Aug 2010 21:36:22 +0000 (UTC) Subject: [Numpy-discussion] Github migration? References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Mon, 23 Aug 2010 23:31:14 +0200, St?fan van der Walt wrote: [clip] > Erk. What's the quickest route to go: compare the actual patches, or > bring a tree up to date for each revision and compute some sort of > working-copy checksum? Working-copy checksumming is probably the easiest and most robust route. svn-all-fast-export adds revision numbers to the commit log messages, so the results should be fairly simple to check. (So the check must be done before these revision numbers are stripped, or in "numpy.save".) Pauli From jsseabold at gmail.com Mon Aug 23 18:28:01 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 23 Aug 2010 18:28:01 -0400 Subject: [Numpy-discussion] DataArray usage question + bug? Message-ID: I have some new "typical" data that I'm trying out DataArray with, and I'm trying to get my head around it again. Is this the best way to hold data organized by, say, household and time (week number). I guess what I'm asking is do I understand the concept of axes and ticks correctly? It seems to be what I want, but my earlier understanding of how I would go about this was a bit different. import numpy as np from datarray.datarray import DataArray ddd = np.empty((52*5,5)) # Household ID ddd[:,0] = np.repeat([1111,2222,3333,4444,5555], 52) # Week Number ddd[:,1] = np.tile(range(1,53), 5) # Some Categorical Variable ddd[:,2] = np.tile(range(1,6), 52) # Some Variable ddd[:,3] = np.random.uniform(0,5.0, size=52*5) # Some Other Variable ddd[:,4] = np.random.uniform(-5,0., size=52*5) # Create axes labels and ticks hhold_ax = 'households', np.unique(ddd[:,0]).tolist() time_ax = 'week', np.unique(ddd[:,1]).tolist() var_ax = 'variables', ['some_dummy', 'some_var', 'other_var'] darr = DataArray(ddd[:,2:].reshape(5,52,-1), [hhold_ax, time_ax, var_ax]) It might be nice to have some convenience functions that will do the reshape on the original 2d data if they don't already exist, so end-users don't have to think about it. As for the bug report. If I don't tolist() the ticks above there is an error. I can file a bug report if it's warranted. --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/skipper/school/RA/ in () /usr/local/lib/python2.6/dist-packages/datarray/datarray.pyc in __new__(cls, data, labels, dtype, copy) 604 axes.append(Axis(label, i, arr, ticks=ticks)) 605 --> 606 _set_axes(arr, axes) 607 608 # validate the axes /usr/local/lib/python2.6/dist-packages/datarray/datarray.pyc in _set_axes(dest, in_axes) 475 # Create the containers for various axis-related info 476 for ax in in_axes: --> 477 new_ax = ax._copy(parent_arr=dest) 478 axes.append(new_ax) 479 if hasattr(ax_holder, ax.name): /usr/local/lib/python2.6/dist-packages/datarray/datarray.pyc in _copy(self, **kwargs) 113 ticks = kwargs.pop('ticks', copy.copy(self.ticks)) 114 ax.ticks = ticks --> 115 if ticks and len(ticks) != len(self.ticks): 116 ax._tick_dict = dict( zip(ticks, xrange( len(ticks) )) ) 117 else: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Skipper From kwgoodman at gmail.com Mon Aug 23 18:49:38 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 23 Aug 2010 15:49:38 -0700 Subject: [Numpy-discussion] DataArray usage question + bug? In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 3:28 PM, Skipper Seabold wrote: > hhold_ax = 'households', np.unique(ddd[:,0]).tolist() > As for the bug report. ?If I don't tolist() the ticks above there is > an error. ?I can file a bug report if it's warranted. If you add it to the tracker (http://github.com/fperez/datarray/issues) then I am sure it will get discussed at the next sprint. From fperez.net at gmail.com Mon Aug 23 21:17:55 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 23 Aug 2010 18:17:55 -0700 Subject: [Numpy-discussion] Datarray sprint July 28, 2010 In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 2:19 PM, Keith Goodman wrote: > On Mon, Aug 23, 2010 at 2:17 PM, Skipper Seabold wrote: >> Is Fernando's github still the most up to date location for datarray? > > Yes. We'd love to move it out of my page to a numpy-owned repo, so once that dust settles I'm more than happy to ask github for a move. It makes no real sense that it's in my github account, as I have no more special rights in there than anyone else (and likely less time available to do real work than others). Since it seems that we're now very close to a real move of numpy to github, we should be good to move this over in a week or two. Regards, f From charlesr.harris at gmail.com Mon Aug 23 22:54:30 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Aug 2010 20:54:30 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Mon, Aug 23, 2010 at 2:30 PM, Travis Oliphant wrote: > > Hi all, > > I'm curious as to the status of the Github migration and if there is > anything I can do to help. I have a couple of weeks right now and I would > love to see us make the transition of both NumPy and SciPy to GIT. > > On a slightly related note, it would really help the numpy-refactor project > if it received more input from others in the community. Right now, the > numpy-refactor is happening on a public github branch (named numpy-refactor) > awaiting numpy itself to be on github. It would be more useful if the > numpy-refactor branch were a regular branch of the github NumPy project. > > The numpy-refactor project is making great progress and we have a working > core library that can be built on Windows, Mac, and Linux. The goal is > for this numpy-refactor to become the basis for NumPy 2.0 which should come > out sometime this Fall. Already, a lot of unit tests have been written > and code coverage has increased on the core NumPy code which I think we all > agree is "a good thing" In addition, some lingering reference count bugs > (particularly in object arrays) have been found and squashed. > > There is also some good progress on the Cython backend for .NET which would > allow and also put pressure on migration of most of SciPy to Cython-or-Fwrap > generated extension modules. > > I am looking forward to working on NumPy a little more over the coming > months. > > All the best, > > I've been having some fun browsing through the branch but don't have much to say on such short acquaintance. I wonder if the patch in ticket #1085might be something you folks could look at now that the loops have been moved about and such? Also, it would be nice if the extended comment style was rigidly adhered to, although things look pretty good in that department. Another nit would be to keep an eye out during the cleanups for "if (blah) foo;" if statements and clean them up by putting the foo on a separate line when it is convenient to do so. Apart from that it looks like Ilan and Jason are really getting into it and doing a nice job of regularizing the naming conventions and such which should make the code easier to read and maintain. Adding some explanatory comments along the way would also help as it may be awhile before someone else looks so closely at the code as to have a good understanding of what the different functions do. I know that gets in the way of getting the code out, but I throw it out as a wish. Would it be possible to sketch out how the general layering of functionality looks at this point? Is there going to be any cleanup of the visible macros? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Aug 23 23:22:08 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 23 Aug 2010 22:22:08 -0500 Subject: [Numpy-discussion] further to my previous numpy installation problems In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 3:57 PM, martin djokovic wrote: > First of all sorry about not posting as a reply in the above thread-for some > reason it does not seem to work > > So have to post as a new question-sorry about that. > > Ok so got a tar file installed as root, then did > > LOCATE NUMPY > > All most all the files found with numpy were in > > /usr/local/lib/python2.4/site-packages/numpy > > Which is where I installed numpy......... > > Both root and user are set to use python 2.4 > > However the following were also listed- > > I as root can remove them but I am scared that that might mess something > else-should I remove them and try again? I am specially concerned about the > redhat/SOURCES and also the first entry (which I did remove before > installing this time but has come back) > > /usr/lib/python2.4/site-packages/numpy > > /usr/local/share/pymol/ext/lib/python2.4/site-packages/Numeric/_numpy.so > > /usr/share/doc/numpy-1.0.3 > > /usr/src/redhat/SOURCES/numpy-1.0.1-f2py.patch > > /usr/src/redhat/SOURCES/numpy-1.0.3-2.tar.gz > > /usr/src/redhat/SOURCES/numpy-1.0.3-setup.patch > > /usr/src/redhat/SPECS/numpy.spec > > /scratch/numpy-1.0.4 > > /usr/bin/f2py.numpy > > /usr/lib/python2.4/site-packages/Numeric/_numpy.so > > /usr/local/biopython-1.53/numpy > > /usr/local/lib/python2.4/site-packages/Numeric/_numpy.so > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > So you have two python2.4 installations: 1) /usr/lib/python2.4 that is from Fedora (because you have the redhat sources directory) and is not used (but could be via full path '/usr/bin/python'. 2) /usr/local/lib/python2.4 The only way that I can explain that is that you installed your own version of python2.4 and perhaps later the distro updated it's version. Or something similar. Reverting back to the distro version is not that easy but reinstalling Python could do it. However, you would need either the original disks or point your package manager to use the appropriate repositories (some are linked available at http://fedoralegacy.org/download/fedoralegacy-mirrors.php ). You might that those disks or links if you do not have what you need for scipy (Fortran compiler with the LAPACK, BLAS, and ATLAS libraries). Anyhow you appear to be using the /usr/local stuff so you should not touch that unless you want to reinstall numpy again. I would suggest trying to use the package manager or yum/rpm to remove the numpy package and for that matter Numeric. That should actually remove most of these entries and leave you in 'control' of numpy. You might find that there may be other dependencies but it should not touch anything in the '/usr/local' directory. Bruce From charlesr.harris at gmail.com Mon Aug 23 23:30:55 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Aug 2010 21:30:55 -0600 Subject: [Numpy-discussion] Polynomials Message-ID: Hi All, I've gone ahead and implemented the Laguerre and Hermite (H and He) polynomials, but at this point I'm loath to add them to numpy as the polynomial space is getting crowded. Scipy looks like a better spot to put these but there is already the orthogonal module there. OTOH, the orthogonal module uses poly1d for the representation and that is numerical poison. The versions I have use the direct series representations in the given basis for +, -, x, /, integration, and differentiation as well as doing direct conversions between domains and basis without going through ordinary power series, so that is a plus. They are also usable with mpmath for extended precision but at the price of sticking to python for the implementation, so they could be speeded up by restricting to doubles and complex. They don't supply the points and weights for Gauss integration, I tend to think those things belong under integration, but they could be included. Anyway, I'm looking to restart the polynomial discussion before doing something drastic so would appreciate any input. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Aug 23 23:38:36 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Aug 2010 21:38:36 -0600 Subject: [Numpy-discussion] further to my previous numpy installation problems In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 9:22 PM, Bruce Southey wrote: > On Mon, Aug 23, 2010 at 3:57 PM, martin djokovic > wrote: > > First of all sorry about not posting as a reply in the above thread-for > some > > reason it does not seem to work > > > > So have to post as a new question-sorry about that. > > > > Ok so got a tar file installed as root, then did > > > > LOCATE NUMPY > > > > All most all the files found with numpy were in > > > > /usr/local/lib/python2.4/site-packages/numpy > > > > Which is where I installed numpy......... > > > > Both root and user are set to use python 2.4 > > > > However the following were also listed- > > > > I as root can remove them but I am scared that that might mess something > > else-should I remove them and try again? I am specially concerned about > the > > redhat/SOURCES and also the first entry (which I did remove before > > installing this time but has come back) > > > > /usr/lib/python2.4/site-packages/numpy > > > > /usr/local/share/pymol/ext/lib/python2.4/site-packages/Numeric/_numpy.so > > > > /usr/share/doc/numpy-1.0.3 > > > > /usr/src/redhat/SOURCES/numpy-1.0.1-f2py.patch > > > > /usr/src/redhat/SOURCES/numpy-1.0.3-2.tar.gz > > > > /usr/src/redhat/SOURCES/numpy-1.0.3-setup.patch > > > > /usr/src/redhat/SPECS/numpy.spec > > > > /scratch/numpy-1.0.4 > > > > /usr/bin/f2py.numpy > > > > /usr/lib/python2.4/site-packages/Numeric/_numpy.so > > > > /usr/local/biopython-1.53/numpy > > > > /usr/local/lib/python2.4/site-packages/Numeric/_numpy.so > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > So you have two python2.4 installations: > 1) /usr/lib/python2.4 that is from Fedora (because you have the > redhat sources directory) and is not used (but could be via full path > '/usr/bin/python'. > 2) /usr/local/lib/python2.4 > > The only way that I can explain that is that you installed your own > version of python2.4 and perhaps later the distro updated it's > version. Or something similar. Reverting back to the distro version is > not that easy but reinstalling Python could do it. However, you would > need either the original disks or point your package manager to use > the appropriate repositories (some are linked available at > http://fedoralegacy.org/download/fedoralegacy-mirrors.php ). You might > that those disks or links if you do not have what you need for scipy > (Fortran compiler with the LAPACK, BLAS, and ATLAS libraries). > > Anyhow you appear to be using the /usr/local stuff so you should not > touch that unless you want to reinstall numpy again. > > I would suggest trying to use the package manager or yum/rpm to remove > the numpy package and for that matter Numeric. That should actually > remove most of these entries and leave you in 'control' of numpy. You > might find that there may be other dependencies but it should not > touch anything in the '/usr/local' directory. > > I would leave the Fedora Python alone as Fedora uses it and taking it out will delete various modules needed by yum and other interface bits. Restoring functionality after that can be a pain and, yes, I made that mistake once. I would stick to the system python just for simplicity. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Aug 24 01:30:18 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 24 Aug 2010 07:30:18 +0200 Subject: [Numpy-discussion] Datarray sprint July 28, 2010 In-Reply-To: References: Message-ID: On 24 August 2010 03:17, Fernando Perez wrote: > We'd love to move it out of my page to a numpy-owned repo, so once > that dust settles I'm more than happy to ask github for a move. I guess we can just create the new repo ourselves, or does GitHub track some other meta-data that should be moved? Regards St?fan From fperez.net at gmail.com Tue Aug 24 03:59:24 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 24 Aug 2010 00:59:24 -0700 Subject: [Numpy-discussion] Datarray sprint July 28, 2010 In-Reply-To: References: Message-ID: 2010/8/23 St?fan van der Walt : > On 24 August 2010 03:17, Fernando Perez wrote: >> We'd love to move it out of my page to a numpy-owned repo, so once >> that dust settles I'm more than happy to ask github for a move. > > I guess we can just create the new repo ourselves, or does GitHub > track some other meta-data that should be moved? Yes, please do *not* create it. We need to make a manual move request, see for example: http://support.github.com/discussions/organization-issues/123-request-to-move-a-personal-repo-to-an-organization This ensures that all metadata moves with the repo (issues, fork queues, etc). Once moved, only I have to re-fork and we're done. People will then need to update their remote urls in their clones (or re-clone if they are lazy) but that's it. Cheers, f From asmund.hjulstad at gmail.com Tue Aug 24 07:16:09 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Tue, 24 Aug 2010 14:16:09 +0300 Subject: [Numpy-discussion] Evaluating performance of f2py extensions with gprof, why spending time _gfortran_compare_string In-Reply-To: References: Message-ID: 2010/8/18 ?smund Hjulstad > > I am calling a few functions in a fortran library. All parameters are short > (longest array of 20 elements), and I do three calls to the fortran library > pr iteration. According to the python profiler (running the script as %run > -p in ipython), all time is spent in the python extension. > > I built the extension with options -pg -O , ran a test script, and > evaluated the output with > > gprof .py -b > > with the following output: > > Flat profile: > > Each sample counts as 0.01 seconds. > % cumulative self self total > time seconds seconds calls Ts/call Ts/call name > 41.64 5.03 5.03 > _gfortran_compare_string > 27.40 8.34 3.31 rdxhmx_ > I have found one gotcha in my approach, and that is that I have been building the extension using distutils. It appears that extra_compile_args options are not passed to fortran compilers (only c compilers). As a result, the call graph was missing most of the interesting information. After a change in GnuFCompiler, I now force distutils to compile fortran codes with -pg enabled. Perhaps this should be filed as a feature request? (distutils compilation with profiling enabled?, or at least the option of passing additional arguments to gfortran) Back to my original problem, though, I'm still struggling. What is calling _gfortran_compare_string, that still is taking up more than 40% of processing time? ----------------------------------------------- [6] 43.1 5.10 0.00 _gfortran_compare_string [6] ----------------------------------------------- Any suggestions? -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Aug 24 10:58:54 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 24 Aug 2010 09:58:54 -0500 Subject: [Numpy-discussion] Polynomials In-Reply-To: References: Message-ID: <4E68AC23-D84C-4D9E-B468-88D7B1B82E36@enthought.com> On Aug 23, 2010, at 10:30 PM, Charles R Harris wrote: > Hi All, > > I've gone ahead and implemented the Laguerre and Hermite (H and He) polynomials, but at this point I'm loath to add them to numpy as the polynomial space is getting crowded. Scipy looks like a better spot to put these but there is already the orthogonal module there. OTOH, the orthogonal module uses poly1d for the representation and that is numerical poison. The versions I have use the direct series representations in the given basis for +, -, x, /, integration, and differentiation as well as doing direct conversions between domains and basis without going through ordinary power series, so that is a plus. They are also usable with mpmath for extended precision but at the price of sticking to python for the implementation, so they could be speeded up by restricting to doubles and complex. They don't supply the points and weights for Gauss integration, I tend to think those things belong under integration, but they could be included. Don't let the orthogonal module stop you from adding the polynomials to SciPy. They would fit very nicely. It would be nice to deprecate the current orthogonal polynomials over time as more numerically accurate implementations of them were included. You could put them in a module like: scipy.special.ortho or another name. You could even replace the laguerre and hermite functions in special.orthogonal assuming the behaviors are similar. But, to be safe, it would probably be best to put them in a different module and deprecate the old names. Thanks for everything. -Travis From seb.haase at gmail.com Tue Aug 24 21:39:54 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 25 Aug 2010 03:39:54 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Sun, Aug 22, 2010 at 10:35 AM, Robin wrote: > On Sun, Aug 22, 2010 at 8:41 AM, Sebastian Haase wrote: >>>>> Do you know if that contains a C++ compiler ? ?The first page before >>>>> it starts the actual download has "Visual C++ Compilers" grayed out >>>>> ... !? >>>>> >>>>> -Sebastian >>>>> >>>> Ok, apparently I had to install the "dot NET Framework 4" from >>>> http://msdn.microsoft.com/en-us/netframework/aa569263.aspx >>>> first, before then the C++ could be installed. >>>> But now setup.py still complains: >>>> ? ? ? error: Unable to find vcvarsall.bat >>>> and I think it is looking for >>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0 >>>> while that file got installed in >>>> C:\Program Files (x86)\Microsoft Visual Studio 10.0 >>>> I don't know how to get the "log.debug" messages from the setup.py >>>> script activated... >>>> ? > > My limited understanding is that there are two different versions of > microsoft stuff at the moment: > VS 2008 which goes with SDK 3.5 and VS 2010 which goes with SDK 4.0. I > think Python is built with 2008 so that might be easier to try. I'm > not sure though. > > While the SDK contains some compilers I think the easiest way to get > the compilers working is to first install the VS express edition, > which is 32 bit, and then install the SDK which provides 64 bit > support. Here are some instructions I lifted from the Matlab website > (I haven't tried building Python extensions yet but I was able to > embed Python with this setup). > > To install Visual Studio 2008 Express Edition with all required components: > 1. Install Microsoft Visual Studio 2008 Express Edition. The main > Visual Studio 2008 Express installer is available from: > http://www.microsoft.com/express/Downloads/#Visual_Studio_2008_Express_Downloads > This package can be installed using the default options. > > 2. Install the Microsoft Windows SDK. The Microsoft Windows SDK is > available by searching Microsoft's download site, or by going directly > to: > http://msdn.microsoft.com/en-us/windowsserver/bb980924.aspx > or > http://www.microsoft.com/downloads/details.aspx?FamilyId=F26B1AA4-741A-433A-9BE5-FA919850BDBF&displaylang=en > > Download the Windows Server 2008 & .NET 3.5 SDK. Do not install beta > or 'Release Candidate' (RC) versions. Also do NOT install "Microsoft > Windows SDK for Windows 7 and .NET Framework 4" (version 7.1); if you > want to use a 7.x version choose the "Microsoft Windows SDK for > Windows 7 and .NET Framework 3.5 SP1". > > 2.1. While installing the SDK, you must select "x64 Compilers and > Tools". For example, in the SDK installer above: > On screen "Installation Options" > Select "Developer Tools"->"Visual C++ Compilers". > This item has the Feature Description "Install the Visual C++ 9.0 > Compilers. These compilers allow you to target x86, x64, IA64 > processor architectures." > > 3. To verify that you have all installed components, check that the > Microsoft SDK contains the "amd64" version of the C/C++ compiler > "cl.exe". This is usually installed into > > > C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\cl.exe > > > Cheers > > Robin Robin, thanks for those links. My experience is more like the one described by fuzion at http://nukeit.org/compile-python-2-7-packages-with-visual-studio-2010-express/ That is, I did not see any problem - yet. But since I blindly believe you guys here ... I will now research on how de-install the .NET 4 and the VS 2010 stuff.... -Sebastian From cournape at gmail.com Tue Aug 24 22:02:29 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Aug 2010 11:02:29 +0900 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Wed, Aug 25, 2010 at 10:39 AM, Sebastian Haase wrote: > Robin, > thanks for those links. > My experience is more like the one described by fuzion at > http://nukeit.org/compile-python-2-7-packages-with-visual-studio-2010-express/ > That is, I did not see any problem - yet. The problem is that when you will have issues, they will be painful and hard to debug. For example, you will get a file descriptor from some part of C code using one CRT, and will try to access it in some another part using read/write from another CRT, and it will not work at all. Given that using the SDK for 2008 is easier than the one for 2010 anyway, I don't see any reason not to use the recommended one when developing against python2.6/2.7. cheers, David From seb.haase at gmail.com Tue Aug 24 22:06:42 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 25 Aug 2010 04:06:42 +0200 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Wed, Aug 25, 2010 at 4:02 AM, David Cournapeau wrote: > On Wed, Aug 25, 2010 at 10:39 AM, Sebastian Haase wrote: > >> Robin, >> thanks for those links. >> My experience is more like the one described by fuzion at >> http://nukeit.org/compile-python-2-7-packages-with-visual-studio-2010-express/ >> That is, I did not see any problem - yet. > > The problem is that when you will have issues, they will be painful > and hard to debug. For example, you will get a file descriptor from > some part of C code using one CRT, and will try to access it in some > another part using read/write from another CRT, and it will not work > at all. Given that using the SDK for 2008 is easier than the one for > 2010 anyway, I don't see any reason not to use the recommended one > when developing against python2.6/2.7. > > cheers, > > David Thanks for the insight. One more: Is python 3.1 + 3.2 then to be used with VS 2010 - i.e. msvcr100? -S. From cournape at gmail.com Tue Aug 24 22:09:55 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Aug 2010 11:09:55 +0900 Subject: [Numpy-discussion] mingw-w64 tutorial ? In-Reply-To: References: <4C704524.2010303@uci.edu> <4C704CED.2030602@uci.edu> Message-ID: On Wed, Aug 25, 2010 at 11:06 AM, Sebastian Haase wrote: > Thanks for the insight. More experience than insight ;) > One more: ?Is python 3.1 + 3.2 then to be used with VS 2010 ?- i.e. msvcr100? I believe no python binary is built with VS 2010. MvL is generally cautious about using new compilers, but you can and should check by yourself using the sys module (sys.version for example returns the MSVC version, and in general, you can associate it to the CRT version. VS 2008 is MSVC 15 with msvcr9 as in python 2.6 and 2.7). cheers, David From dagss at student.matnat.uio.no Wed Aug 25 04:04:50 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 25 Aug 2010 10:04:50 +0200 Subject: [Numpy-discussion] ANN: Cython 0.13 released! Message-ID: <4C74CEA2.1070703@student.matnat.uio.no> Forwarding; Crag Citro wrote: It is with *great* pleasure that I email to announce the release of Cython version 0.13! This release sets another milestone on the path towards Python compatibility and brings major new features and improvements for the usability of the Cython language. Download it here: http://cython.org/release/Cython-0.13.tar.gz == New Features == * Closures are fully supported for Python functions. Cython supports inner functions and lambda expressions. Generators and generator expressions are __not__ supported in this release. * Proper C++ support. Cython knows about C++ classes, templates and overloaded function signatures, so that Cython code can interact with them in a straight forward way. * Type inference is enabled by default for safe C types (e.g. double, bint, C++ classes) and known extension types. This reduces the need for explicit type declarations and can improve the performance of untyped code in some cases. There is also a verbose compile mode for testing the impact on user code. * Cython's for-in-loop can iterate over C arrays and sliced pointers. The type of the loop variable will be inferred automatically in this case. * The Py_UNICODE integer type for Unicode code points is fully supported, including for-loops and 'in' tests on unicode strings. It coerces from and to single character unicode strings. Note that untyped for-loop variables will automatically be inferred as Py_UNICODE when iterating over a unicode string. In most cases, this will be much more efficient than yielding sliced string objects, but can also have a negative performance impact when the variable is used in a Python context multiple times, so that it needs to coerce to a unicode string object more than once. If this happens, typing the loop variable as unicode or object will help. * The built-in functions any(), all(), sum(), list(), set() and dict() are inlined as plain `for` loops when called on generator expressions. Note that generator expressions are not generally supported apart from this feature. Also, tuple(genexpr) is not currently supported - use tuple([listcomp]) instead. * More shipped standard library declarations. The python_* and stdlib/stdio .pxd files have been deprecated in favor of clib.* and cpython[.*] and may get removed in a future release. == Python compatibility == * Pure Python mode no longer disallows non-Python keywords like 'cdef', 'include' or 'cimport'. It also no longer recognises syntax extensions like the for-from loop. * Parsing has improved for Python 3 syntax in Python code, although not all features are correctly supported. The missing Python 3 features are being worked on for the next release. * from __future__ import print_function is supported in Python 2.6 and later. Note that there is currently no emulation for earlier Python versions, so code that uses print() with this future import will require at least Python 2.6. * New compiler directive language_level (valid values: 2 or 3) with corresponding command line options -2 and -3 requests source code compatibility with Python 2.x or Python 3.x respectively. Language level 3 currently enforces unicode literals for unprefixed string literals, enables the print function (requires Python 2.6 or later) and keeps loop variables in list comprehensions from leaking. * Loop variables in set/dict comprehensions no longer leak into the surrounding scope (following Python 2.7). List comprehensions are unchanged in language level 2. == Incompatible changes == * The availability of type inference by default means that Cython will also infer the type of pointers on assignments. Previously, code like this cdef char* s = ... untyped_variable = s would convert the char* to a Python bytes string and assign that. This is no longer the case and no coercion will happen in the example above. The correct way of doing this is through an explicit cast or by typing the target variable, i.e. cdef char* s = ... untyped_variable1 = s untyped_variable2 = s cdef object py_object = s cdef bytes bytes_string = s * bool is no longer a valid type name by default. The problem is that it's not clear whether bool should refer to the Python type or the C++ type, and expecting one and finding the other has already led to several hard-to-find bugs. Both types are available for importing: you can use from cpython cimport bool for the Python bool type, and from libcpp cimport bool for the C++ type. == Contributors == Many people contributed to this release, including: * David Barnett * Stefan Behnel * Chuck Blake * Robert Bradshaw * Craig Citro * Bryan Cole * Lisandro Dalcin * Eric Firing * Danilo Freitas * Christoph Gohlke * Dag Sverre Seljebotn * Kurt Smith * Erik Tollerud * Carl Witty -cc _______________________________________________ Cython-dev mailing list Cython-dev at codespeak.net http://codespeak.net/mailman/listinfo/cython-dev From stefan at sun.ac.za Wed Aug 25 04:30:16 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 25 Aug 2010 10:30:16 +0200 Subject: [Numpy-discussion] BUG: NumPy exposes the wrong include directory In-Reply-To: References: Message-ID: Hi all, Without this patch, numpy.distutils is broken on, amongs others, Ubuntu. This prohibits packages such as Enthought's Enable from being built successfully. Would anyone object if I applied it to trunk? Regards St?fan 2010/5/5 St?fan van der Walt : > Hi all, > > Under Ubuntu, the NumPy headers are dumped under > /usr/include/python2.6. ?This is not the location that should be > exposed when another copy of NumPy is installed. ?The attached patch > lets > > from numpy.distutils.system_info import get_info > print get_info['numpy'] > > return the correct path. ?However, it's a quick hack, and I'd like > someone familiar with numpy.distutils to have a look and suggest a > proper fix. > > Regards > St?fan > -------------- next part -------------- A non-text attachment was scrubbed... Name: system_info.patch Type: text/x-diff Size: 734 bytes Desc: not available URL: From boolegue at gmail.com Wed Aug 25 04:44:57 2010 From: boolegue at gmail.com (Antoine Dechaume) Date: Wed, 25 Aug 2010 10:44:57 +0200 Subject: [Numpy-discussion] genfromtxt usage Message-ID: Hello, I am trying to read a file with a variable number of values on each lines, using genfromtxt and missing_values or filling_values arguments. The usage of those arguments is not clear in the documentation, if what I am trying to do is possible, how could I do it? Thanks, Antoine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Aug 25 05:02:12 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 25 Aug 2010 11:02:12 +0200 Subject: [Numpy-discussion] genfromtxt usage In-Reply-To: References: Message-ID: Hi Antoine On 25 August 2010 10:44, Antoine Dechaume wrote: > Hello, > I am trying to read a file with a variable number of values on each lines, > using genfromtxt and missing_values or filling_values arguments. > The usage of those arguments is not clear in the documentation, if what I am > trying to do is possible, how could I do it? Neither loadtxt nor genfromtxt were written with a variable number of values in mind. The result is returned as an array, and therefore has to have a set number of columns. In this case, you may be better off writing your own reader, or padding your lines with zeros as appropriate. Regards St?fan From cournape at gmail.com Wed Aug 25 06:58:17 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Aug 2010 19:58:17 +0900 Subject: [Numpy-discussion] BUG: NumPy exposes the wrong include directory In-Reply-To: References: Message-ID: 2010/8/25 St?fan van der Walt : > Hi all, > > Without this patch, numpy.distutils is broken on, amongs others, > Ubuntu. ?This prohibits packages such as Enthought's Enable from being > built successfully. > > Would anyone object if I applied it to trunk? Could you wait a bit more - I would like to look at it in detail tonight. The ubuntu thing is annoying, but ultimately, it is a bug in how we do things, cheers, David From ralf.gommers at googlemail.com Wed Aug 25 10:00:37 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 25 Aug 2010 22:00:37 +0800 Subject: [Numpy-discussion] Polynomials In-Reply-To: <4E68AC23-D84C-4D9E-B468-88D7B1B82E36@enthought.com> References: <4E68AC23-D84C-4D9E-B468-88D7B1B82E36@enthought.com> Message-ID: On Tue, Aug 24, 2010 at 10:58 PM, Travis Oliphant wrote: > > On Aug 23, 2010, at 10:30 PM, Charles R Harris wrote: > > > Hi All, > > > > I've gone ahead and implemented the Laguerre and Hermite (H and He) > polynomials, but at this point I'm loath to add them to numpy as the > polynomial space is getting crowded. Scipy looks like a better spot to put > these but there is already the orthogonal module there. OTOH, the orthogonal > module uses poly1d for the representation and that is numerical poison. The > versions I have use the direct series representations in the given basis for > +, -, x, /, integration, and differentiation as well as doing direct > conversions between domains and basis without going through ordinary power > series, so that is a plus. They are also usable with mpmath for extended > precision but at the price of sticking to python for the implementation, so > they could be speeded up by restricting to doubles and complex. They don't > supply the points and weights for Gauss integration, I tend to think those > things belong under integration, but they could be included. > > > Don't let the orthogonal module stop you from adding the polynomials to > SciPy. They would fit very nicely. It would be nice to deprecate the > current orthogonal polynomials over time as more numerically accurate > implementations of them were included. > > You could put them in a module like: > > scipy.special.ortho > > or another name. You could even replace the laguerre and hermite > functions in special.orthogonal assuming the behaviors are similar. But, to > be safe, it would probably be best to put them in a different module and > deprecate the old names. > > Is your implementation along the same lines as the Polynomial and Chebyshev classes? In that case I would find it odd to split them between numpy and scipy, and perhaps a little arbitrary to have Chebyshev in numpy while Laguerre/Hermite (and possibly others in the future?) go into scipy. And a practical concern may be that if you happen to find bugs or want to make incompatible changes you have to deal with two release cycles. Another option to consider would be to stick the low-level functions in a separate namespace, and just have the main classes in numpy.polynomial. Then your concern about the crowded namespace is gone, and you could add your new code to numpy. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmccampbell at enthought.com Wed Aug 25 10:19:45 2010 From: jmccampbell at enthought.com (Jason McCampbell) Date: Wed, 25 Aug 2010 09:19:45 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Chuck, I will update the wiki page on the Numpy developer site that discusses the refactoring this week. Right now what's there reflects our plans before they met the reality of code. Needless to say, the actual implementation differs in some of the details. Here is a very brief overview of the structure: - The libndarray directory now contains all of the code for the 'core' library. This library is independent of Python and implements most of the array, descriptor, iterator, and ufunc functionality. The goal is that all non-trivial behavior should be in here, but in reality some parts are tied fairly tightly to the CPython interpreter and will take more work to move into the core. - numpy/core/src/multiarray and numpy/core/src/umath now implement "just" the CPython interface to libndarray. We have preserved both the Python interface and the C API. Ideally each C API function is just a simple wrapper around a call to the core API, though it doesn't always work out that way. However, a large amount of code has been moved out of these modules into the core. - The core is built as a shared library that is independent of any given interface layer. That is, the same shared library/DLL can be used with CPython, IronPython and any other implementation. Each interface is required to pass in a set of callbacks for handling reference counting, object manipulation, and other interface-specific behavior. - The core implements its own reference counting semantics that happen to look very much like CPython's. This was necessary to make the core library independent of the interface layer and preserve performance (ie, limit the number of callbacks). The handshaking between interface and core is a bit complicated but ends up working nicely and efficiently for both reference counted and garbage collected systems. I'll write up the details on the wiki page. - As Travis mentioned we are also working on a .NET back end to Cython. This lets us port the modules such as MTRAND without having to have two separate interfaces, a Cython and a .NET version. Instead, we can modify the existing .pyx file to use the new core API (should improve performance in CPython version slightly). Once done, Cython can generate the .NET and CPython interfaces from the same .pyx file. We have done a fair amount of cleanup on the naming conventions but certainly more needs to be done! I'll write it up for everyone this week but feel free to email me with other questions. Regards, Jason On Mon, Aug 23, 2010 at 9:54 PM, Charles R Harris wrote: > > > On Mon, Aug 23, 2010 at 2:30 PM, Travis Oliphant wrote: > >> >> Hi all, >> >> I'm curious as to the status of the Github migration and if there is >> anything I can do to help. I have a couple of weeks right now and I would >> love to see us make the transition of both NumPy and SciPy to GIT. >> >> On a slightly related note, it would really help the numpy-refactor >> project if it received more input from others in the community. Right now, >> the numpy-refactor is happening on a public github branch (named >> numpy-refactor) awaiting numpy itself to be on github. It would be more >> useful if the numpy-refactor branch were a regular branch of the github >> NumPy project. >> >> The numpy-refactor project is making great progress and we have a working >> core library that can be built on Windows, Mac, and Linux. The goal is >> for this numpy-refactor to become the basis for NumPy 2.0 which should come >> out sometime this Fall. Already, a lot of unit tests have been written >> and code coverage has increased on the core NumPy code which I think we all >> agree is "a good thing" In addition, some lingering reference count bugs >> (particularly in object arrays) have been found and squashed. >> >> There is also some good progress on the Cython backend for .NET which >> would allow and also put pressure on migration of most of SciPy to >> Cython-or-Fwrap generated extension modules. >> >> I am looking forward to working on NumPy a little more over the coming >> months. >> >> All the best, >> >> > I've been having some fun browsing through the branch but don't have much > to say on such short acquaintance. > > I wonder if the patch in ticket #1085might be something you folks could look at now that the loops have been > moved about and such? Also, it would be nice if the extended comment style > was rigidly adhered to, although things look pretty good in that department. > Another nit would be to keep an eye out during the cleanups for "if (blah) > foo;" if statements and clean them up by putting the foo on a separate line > when it is convenient to do so. Apart from that it looks like Ilan and Jason > are really getting into it and doing a nice job of regularizing the naming > conventions and such which should make the code easier to read and maintain. > Adding some explanatory comments along the way would also help as it may be > awhile before someone else looks so closely at the code as to have a good > understanding of what the different functions do. I know that gets in the > way of getting the code out, but I throw it out as a wish. > > Would it be possible to sketch out how the general layering of > functionality looks at this point? Is there going to be any cleanup of the > visible macros? > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 25 11:20:40 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 25 Aug 2010 09:20:40 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Wed, Aug 25, 2010 at 8:19 AM, Jason McCampbell wrote: > Chuck, > > I will update the wiki page on the Numpy developer site that discusses the > refactoring this week. Right now what's there reflects our plans before > they met the reality of code. Needless to say, the actual implementation > differs in some of the details. > > Here is a very brief overview of the structure: > > - The libndarray directory now contains all of the code for the 'core' > library. This library is independent of Python and implements most of the > array, descriptor, iterator, and ufunc functionality. The goal is that all > non-trivial behavior should be in here, but in reality some parts are tied > fairly tightly to the CPython interpreter and will take more work to move > into the core. > > - numpy/core/src/multiarray and numpy/core/src/umath now implement "just" > the CPython interface to libndarray. We have preserved both the Python > interface and the C API. Ideally each C API function is just a simple > wrapper around a call to the core API, though it doesn't always work out > that way. However, a large amount of code has been moved out of these > modules into the core. > > - The core is built as a shared library that is independent of any given > interface layer. That is, the same shared library/DLL can be used with > CPython, IronPython and any other implementation. Each interface is > required to pass in a set of callbacks for handling reference counting, > object manipulation, and other interface-specific behavior. > > - The core implements its own reference counting semantics that happen to > look very much like CPython's. This was necessary to make the core library > independent of the interface layer and preserve performance (ie, limit the > number of callbacks). The handshaking between interface and core is a bit > complicated but ends up working nicely and efficiently for both reference > counted and garbage collected systems. I'll write up the details on the > wiki page. > > - As Travis mentioned we are also working on a .NET back end to Cython. > This lets us port the modules such as MTRAND without having to have two > separate interfaces, a Cython and a .NET version. Instead, we can modify > the existing .pyx file to use the new core API (should improve performance > in CPython version slightly). Once done, Cython can generate the .NET and > CPython interfaces from the same .pyx file. > > We have done a fair amount of cleanup on the naming conventions but > certainly more needs to be done! > > I'll write it up for everyone this week but feel free to email me with > other questions. > > Thanks for the summary, it clarifies things a lot. On my cleanup wish list, some of the functions use macros that contain jumps, which is not so nice. I've been intending to scratch that itch for several years now but haven't gotten around to it. I expect such things have a lower priority than getting the basic separation of functionality in place, but just in case... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 25 11:32:11 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 25 Aug 2010 09:32:11 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Wed, Aug 25, 2010 at 8:19 AM, Jason McCampbell wrote: > Chuck, > > I will update the wiki page on the Numpy developer site that discusses the > refactoring this week. Right now what's there reflects our plans before > they met the reality of code. Needless to say, the actual implementation > differs in some of the details. > > Here is a very brief overview of the structure: > > - The libndarray directory now contains all of the code for the 'core' > library. This library is independent of Python and implements most of the > array, descriptor, iterator, and ufunc functionality. The goal is that all > non-trivial behavior should be in here, but in reality some parts are tied > fairly tightly to the CPython interpreter and will take more work to move > into the core. > > - numpy/core/src/multiarray and numpy/core/src/umath now implement "just" > the CPython interface to libndarray. We have preserved both the Python > interface and the C API. Ideally each C API function is just a simple > wrapper around a call to the core API, though it doesn't always work out > that way. However, a large amount of code has been moved out of these > modules into the core. > > - The core is built as a shared library that is independent of any given > interface layer. That is, the same shared library/DLL can be used with > CPython, IronPython and any other implementation. Each interface is > required to pass in a set of callbacks for handling reference counting, > object manipulation, and other interface-specific behavior. > > How do you manage PyCapsule/PyCObject? I don't recall how deeply they were used but ISTR that they were used below the top level interface layer in several places. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Aug 25 11:44:52 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 25 Aug 2010 09:44:52 -0600 Subject: [Numpy-discussion] Polynomials In-Reply-To: References: <4E68AC23-D84C-4D9E-B468-88D7B1B82E36@enthought.com> Message-ID: On Wed, Aug 25, 2010 at 8:00 AM, Ralf Gommers wrote: > > > On Tue, Aug 24, 2010 at 10:58 PM, Travis Oliphant wrote: > >> >> On Aug 23, 2010, at 10:30 PM, Charles R Harris wrote: >> >> > Hi All, >> > >> > I've gone ahead and implemented the Laguerre and Hermite (H and He) >> polynomials, but at this point I'm loath to add them to numpy as the >> polynomial space is getting crowded. Scipy looks like a better spot to put >> these but there is already the orthogonal module there. OTOH, the orthogonal >> module uses poly1d for the representation and that is numerical poison. The >> versions I have use the direct series representations in the given basis for >> +, -, x, /, integration, and differentiation as well as doing direct >> conversions between domains and basis without going through ordinary power >> series, so that is a plus. They are also usable with mpmath for extended >> precision but at the price of sticking to python for the implementation, so >> they could be speeded up by restricting to doubles and complex. They don't >> supply the points and weights for Gauss integration, I tend to think those >> things belong under integration, but they could be included. >> >> >> Don't let the orthogonal module stop you from adding the polynomials to >> SciPy. They would fit very nicely. It would be nice to deprecate the >> current orthogonal polynomials over time as more numerically accurate >> implementations of them were included. >> >> You could put them in a module like: >> >> scipy.special.ortho >> >> or another name. You could even replace the laguerre and hermite >> functions in special.orthogonal assuming the behaviors are similar. But, to >> be safe, it would probably be best to put them in a different module and >> deprecate the old names. >> >> Is your implementation along the same lines as the Polynomial and > Chebyshev classes? In that case I would find it odd to split them between > numpy and scipy, and perhaps a little arbitrary to have Chebyshev in numpy > while Laguerre/Hermite (and possibly others in the future?) go into scipy. > And a practical concern may be that if you happen to find bugs or want to > make incompatible changes you have to deal with two release cycles. > > Yes. It turns out that all that is needed for the arithmetic is a multiply-by-x function which comes from the recursion relationship, and all that is needed for the conversions between types is the evaluation function, which also comes from the recursion relationship. Which leaves integration/differentiation as the difficult bits. Although I could also implement those using the recursion relationship that doesn't seem the best way to go, especially where simpler relationships are available. But maybe I will go that way just for simplicity... Another option to consider would be to stick the low-level functions in a > separate namespace, and just have the main classes in numpy.polynomial. Then > your concern about the crowded namespace is gone, and you could add your new > code to numpy. > > I was mostly concerned about the amount of code, although most of the space is taken up by comments, the code itself tends to be short. The comments are repetitious and mostly consist of cut-and-paste with name substitutions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Wed Aug 25 13:41:37 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 25 Aug 2010 12:41:37 -0500 Subject: [Numpy-discussion] bug in dtype.__eq__ method ? In-Reply-To: <20100823165526.GB23997@in.waw.pl> References: <20100823165009.GA5818@tulpenbaum.cognition.tu-berlin.de> <20100823165526.GB23997@in.waw.pl> Message-ID: On Aug 23, 2010, at 11:55 AM, Zbyszek Szmek wrote: > On Mon, Aug 23, 2010 at 06:50:09PM +0200, Tiziano Zito wrote: >> hi all, >> we just noticed the following weird thing: >> >> $ python >> Python 2.6.6rc2 (r266rc2:84114, Aug 18 2010, 07:33:44) >> [GCC 4.4.5 20100816 (prerelease)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy >>>>> numpy.version.version >> '2.0.0.dev8469' > Also with numpy.version.version == 1.3.0. This certainly looks odd. If you can file a bug-report that would be great. -Travis > > Best, > Zbyszek > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com From zbyszek at in.waw.pl Wed Aug 25 16:14:27 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Wed, 25 Aug 2010 22:14:27 +0200 Subject: [Numpy-discussion] bug in dtype.__eq__ method ? In-Reply-To: References: <20100823165009.GA5818@tulpenbaum.cognition.tu-berlin.de> <20100823165526.GB23997@in.waw.pl> Message-ID: <20100825201427.GC23997@in.waw.pl> On Wed, Aug 25, 2010 at 12:41:37PM -0500, Travis Oliphant wrote: > > On Aug 23, 2010, at 11:55 AM, Zbyszek Szmek wrote: > > > On Mon, Aug 23, 2010 at 06:50:09PM +0200, Tiziano Zito wrote: > >> hi all, > >> we just noticed the following weird thing: > >> > >> $ python > >> Python 2.6.6rc2 (r266rc2:84114, Aug 18 2010, 07:33:44) > >> [GCC 4.4.5 20100816 (prerelease)] on linux2 > >> Type "help", "copyright", "credits" or "license" for more information. > >>>>> import numpy > >>>>> numpy.version.version > >> '2.0.0.dev8469' > > Also with numpy.version.version == 1.3.0. > > This certainly looks odd. If you can file a bug-report that would be great. arraydescr_richcompare calls PyArray_DescrConverter, which converts None to PyArray_DEFAULT... /*NUMPY_API * Get typenum from an object -- None goes to PyArray_DEFAULT * This function takes a Python object representing a type and converts it * to a the correct PyArray_Descr * structure to describe the type. * * Many objects can be used to represent a data-type which in NumPy is * quite a flexible concept. * * This is the central code that converts Python objects to * Type-descriptor objects that are used throughout numpy. * new reference in *at */ NPY_NO_EXPORT int PyArray_DescrConverter(PyObject *obj, PyArray_Descr **at) So: >>> numpy.dtype('float32') == '>> numpy.dtype('float32') == 'float64' False >>> numpy.dtype('float32') == 'float32' True >>> numpy.dtype('float32') == 'float33' Traceback (most recent call last): File "", line 1, in TypeError: data type not understood I think that this automatic conversion is pretty dangerous, especially in case of None. Best, Zbyszek From njs at pobox.com Wed Aug 25 21:16:55 2010 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 25 Aug 2010 18:16:55 -0700 Subject: [Numpy-discussion] curious behavior with indexed assignment Message-ID: So on IRC #scipy just now, Carlos Scheidegger pointed out some behavior that I at least find very surprising: >>> a = np.zeros(10) >>> a[:] = np.arange(3) ValueError: shape mismatch: objects cannot be broadcast to a single shape Okay, that's what we expected. But if we use explicit indices... >>> a[np.arange(10)] = np.arange(3) >>> a array([ 0., 1., 2., 0., 1., 2., 0., 1., 2., 0.]) Also works with boolean masks: >>> a = np.zeros(10) >>> a[np.ones(10, dtype=bool)] = np.arange(3) >>> a array([ 0., 1., 2., 0., 1., 2., 0., 1., 2., 0.]) And if the array being assigned is too short, then the rvalue gets silently truncated: >>> a = np.zeros(10) >>> a[np.arange(5)] = np.arange(10) >>> a array([ 0., 1., 2., 3., 4., 0., 0., 0., 0., 0.]) Shouldn't these all give errors? This "recycling rule" behavior (as R would call it) seems totally inconsistent with usual broadcasting rules, but I can't seem to find any documentation or anything on it. (And indeed, I tried reproducing it with 1-d slices of a 2-d array, and just got "array is not broadcastable to the correct shape" exceptions.) Is this a bug or intentional? (Tested with self-build Numpy 1.4.1 on Python 2.6.5.) -- Nathaniel From stefan at sun.ac.za Thu Aug 26 06:16:05 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 26 Aug 2010 12:16:05 +0200 Subject: [Numpy-discussion] curious behavior with indexed assignment In-Reply-To: References: Message-ID: On 26 August 2010 03:16, Nathaniel Smith wrote: > So on IRC #scipy just now, Carlos Scheidegger pointed out some > behavior that I at least find very surprising: > >>>> a = np.zeros(10) >>>> a[:] = np.arange(3) > ValueError: shape mismatch: objects cannot be broadcast to a single shape Another one you'll find interesting: x.flat = np.arange(3) Cheers St?fan From ralf.gommers at googlemail.com Thu Aug 26 07:30:16 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 26 Aug 2010 19:30:16 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 release candidate 1 Message-ID: I am pleased to announce the availability of the first release candidate of NumPy 1.5.0. This will be the first NumPy release to include support for Python 3, as well as for Python 2.7. Please try this RC and report any problems on the NumPy mailing list. Especially with Python 3 testing will be very useful. On Linux and OS X building from source should be straightforward, for Windows a binary installer is provided. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ Enjoy, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bioinformed at gmail.com Thu Aug 26 10:00:05 2010 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Thu, 26 Aug 2010 10:00:05 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 release candidate 1 In-Reply-To: References: Message-ID: On Thu, Aug 26, 2010 at 7:30 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the first release candidate > of NumPy 1.5.0. This will be the first NumPy release to include support > for Python 3, as well as for Python 2.7. > > Please try this RC and report any problems on the NumPy mailing list. > Especially with Python 3 testing will be very useful. On Linux and OS X > building from source should be straightforward, for Windows a binary > installer is provided. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > > Great to hear! Does this release also add support for building Cython extensions using numpy distutils? The previous hacks I've used no longer work with Python 2.7. Thanks, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmccampbell at enthought.com Thu Aug 26 10:33:33 2010 From: jmccampbell at enthought.com (Jason McCampbell) Date: Thu, 26 Aug 2010 09:33:33 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Wed, Aug 25, 2010 at 10:20 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Wed, Aug 25, 2010 at 8:19 AM, Jason McCampbell < > jmccampbell at enthought.com> wrote: > >> Chuck, >> >> I will update the wiki page on the Numpy developer site that discusses the >> refactoring this week. Right now what's there reflects our plans before >> they met the reality of code. Needless to say, the actual implementation >> differs in some of the details. >> >> Here is a very brief overview of the structure: >> >> - The libndarray directory now contains all of the code for the 'core' >> library. This library is independent of Python and implements most of the >> array, descriptor, iterator, and ufunc functionality. The goal is that all >> non-trivial behavior should be in here, but in reality some parts are tied >> fairly tightly to the CPython interpreter and will take more work to move >> into the core. >> >> - numpy/core/src/multiarray and numpy/core/src/umath now implement "just" >> the CPython interface to libndarray. We have preserved both the Python >> interface and the C API. Ideally each C API function is just a simple >> wrapper around a call to the core API, though it doesn't always work out >> that way. However, a large amount of code has been moved out of these >> modules into the core. >> >> - The core is built as a shared library that is independent of any given >> interface layer. That is, the same shared library/DLL can be used with >> CPython, IronPython and any other implementation. Each interface is >> required to pass in a set of callbacks for handling reference counting, >> object manipulation, and other interface-specific behavior. >> >> - The core implements its own reference counting semantics that happen to >> look very much like CPython's. This was necessary to make the core library >> independent of the interface layer and preserve performance (ie, limit the >> number of callbacks). The handshaking between interface and core is a bit >> complicated but ends up working nicely and efficiently for both reference >> counted and garbage collected systems. I'll write up the details on the >> wiki page. >> >> - As Travis mentioned we are also working on a .NET back end to Cython. >> This lets us port the modules such as MTRAND without having to have two >> separate interfaces, a Cython and a .NET version. Instead, we can modify >> the existing .pyx file to use the new core API (should improve performance >> in CPython version slightly). Once done, Cython can generate the .NET and >> CPython interfaces from the same .pyx file. >> >> We have done a fair amount of cleanup on the naming conventions but >> certainly more needs to be done! >> >> I'll write it up for everyone this week but feel free to email me with >> other questions. >> >> > Thanks for the summary, it clarifies things a lot. On my cleanup wish list, > some of the functions use macros that contain jumps, which is not so nice. > I've been intending to scratch that itch for several years now but haven't > gotten around to it. I expect such things have a lower priority than getting > the basic separation of functionality in place, but just in case... > Yes, I know which ones you are talking about -- both goto's and returns. They have been bugging me, too. A few uses have been fixed, but I would like to clean the rest up. We have also been trying to simplify some of the naming to reduce duplication. > How do you manage PyCapsule/PyCObject? I don't recall how deeply they were > used but ISTR that they were used below the top level interface layer in > several places. > Many or most of the uses of them have been removed. There were several instances where either a PyCapsule or a tuple with fixed content was used and need to be accessed in the core. In this cases we just defined a new struct with the appropriate fields. The capsule types never make it to the core and I'd have to do a search to see where they are even used now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Aug 26 10:42:12 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 26 Aug 2010 22:42:12 +0800 Subject: [Numpy-discussion] BUG: NumPy exposes the wrong include directory In-Reply-To: References: Message-ID: On Wed, Aug 25, 2010 at 6:58 PM, David Cournapeau wrote: > 2010/8/25 St?fan van der Walt : > > Hi all, > > > > Without this patch, numpy.distutils is broken on, amongs others, > > Ubuntu. This prohibits packages such as Enthought's Enable from being > > built successfully. > > > > Would anyone object if I applied it to trunk? > > Ehm, didn't you apply it months ago in r8437? It hasn't broken anything, so no objections here. Cheers, Ralf > Could you wait a bit more - I would like to look at it in detail > tonight. The ubuntu thing is annoying, but ultimately, it is a bug in > how we do things, > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 26 11:09:02 2010 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Aug 2010 08:09:02 -0700 Subject: [Numpy-discussion] questions about a "complicated" user-defined dtype and the ufunc API In-Reply-To: References: Message-ID: On Mon, Aug 23, 2010 at 9:20 AM, Travis Oliphant wrote: > On Aug 22, 2010, at 4:36 PM, Nathaniel Smith wrote: >> Question 2: Am I missing something, or does the ufunc API make this >> impossible? The problem is that a "PyUFuncGenericFunction" doesn't >> have any way to find the dtypes of the arrays that it's working on. >> All of the PyArray_ArrFuncs functions take a pointer to the underlying >> ndarray as an argument, so that when working with a string or void >> array, you can find the actual dtype and figure out (e.g.) the size of >> the objects involved. But ufunc inner loops don't get that, so I guess >> it's just impossible to define a ufunc over variable-sized data, or >> data that you need to be able to see the dtype metadata to interpret? > > Yes, currently that is correct. ? Variable data-types don't work in ufuncs for some subtle reasons. ?But, the changes that allow date-times to work also fix (or will fix) this problem as a side-effect. > > The necessary changes to ufuncs have not been made, yet, however. ? They are planned. ? And, yes, this would allow ufuncs to be used for string equality testing, etc. Okay, thanks. It sounds like I'll need to accomplish what I want in some other way for now, then, but if I get some surplus time somehow then maybe I'll take a crack at the "proper" solution :-). -- Nathaniel From ralf.gommers at googlemail.com Thu Aug 26 11:15:14 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 26 Aug 2010 23:15:14 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 release candidate 1 In-Reply-To: References: Message-ID: On Thu, Aug 26, 2010 at 10:00 PM, Kevin Jacobs < bioinformed at gmail.com> wrote: > On Thu, Aug 26, 2010 at 7:30 AM, Ralf Gommers > wrote: > >> I am pleased to announce the availability of the first release candidate >> of NumPy 1.5.0. This will be the first NumPy release to include support >> for Python 3, as well as for Python 2.7. >> >> Please try this RC and report any problems on the NumPy mailing list. >> Especially with Python 3 testing will be very useful. On Linux and OS X >> building from source should be straightforward, for Windows a binary >> installer is provided. >> >> Binaries, sources and release notes can be found at >> https://sourceforge.net/projects/numpy/files/ >> >> > Great to hear! Does this release also add support for building Cython > extensions using numpy distutils? The previous hacks I've used no longer > work with Python 2.7. > > No, that hasn't happened yet. For the cython code in numpy and scipy, the C sources generated from the cython files are checked into svn and those are built, so cython is not needed for the normal build process. If you need this now, can you start a separate thread and include your previous hacks and the error you're getting now? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Aug 26 11:22:51 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 26 Aug 2010 23:22:51 +0800 Subject: [Numpy-discussion] Polynomials In-Reply-To: References: <4E68AC23-D84C-4D9E-B468-88D7B1B82E36@enthought.com> Message-ID: On Wed, Aug 25, 2010 at 11:44 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Wed, Aug 25, 2010 at 8:00 AM, Ralf Gommers > wrote: > >> >> >> On Tue, Aug 24, 2010 at 10:58 PM, Travis Oliphant > > wrote: >> >>> >>> On Aug 23, 2010, at 10:30 PM, Charles R Harris wrote: >>> >>> > Hi All, >>> > >>> > I've gone ahead and implemented the Laguerre and Hermite (H and He) >>> polynomials, but at this point I'm loath to add them to numpy as the >>> polynomial space is getting crowded. Scipy looks like a better spot to put >>> these but there is already the orthogonal module there. OTOH, the orthogonal >>> module uses poly1d for the representation and that is numerical poison. The >>> versions I have use the direct series representations in the given basis for >>> +, -, x, /, integration, and differentiation as well as doing direct >>> conversions between domains and basis without going through ordinary power >>> series, so that is a plus. They are also usable with mpmath for extended >>> precision but at the price of sticking to python for the implementation, so >>> they could be speeded up by restricting to doubles and complex. They don't >>> supply the points and weights for Gauss integration, I tend to think those >>> things belong under integration, but they could be included. >>> >>> >>> Don't let the orthogonal module stop you from adding the polynomials to >>> SciPy. They would fit very nicely. It would be nice to deprecate the >>> current orthogonal polynomials over time as more numerically accurate >>> implementations of them were included. >>> >>> You could put them in a module like: >>> >>> scipy.special.ortho >>> >>> or another name. You could even replace the laguerre and hermite >>> functions in special.orthogonal assuming the behaviors are similar. But, to >>> be safe, it would probably be best to put them in a different module and >>> deprecate the old names. >>> >>> Is your implementation along the same lines as the Polynomial and >> Chebyshev classes? In that case I would find it odd to split them between >> numpy and scipy, and perhaps a little arbitrary to have Chebyshev in numpy >> while Laguerre/Hermite (and possibly others in the future?) go into scipy. >> And a practical concern may be that if you happen to find bugs or want to >> make incompatible changes you have to deal with two release cycles. >> >> > Yes. It turns out that all that is needed for the arithmetic is a > multiply-by-x function which comes from the recursion relationship, and all > that is needed for the conversions between types is the evaluation function, > which also comes from the recursion relationship. Which leaves > integration/differentiation as the difficult bits. Although I could also > implement those using the recursion relationship that doesn't seem the best > way to go, especially where simpler relationships are available. But maybe I > will go that way just for simplicity... > > Another option to consider would be to stick the low-level functions in a >> separate namespace, and just have the main classes in numpy.polynomial. Then >> your concern about the crowded namespace is gone, and you could add your new >> code to numpy. >> >> > I was mostly concerned about the amount of code, although most of the space > is taken up by comments, the code itself tends to be short. The comments are > repetitious and mostly consist of cut-and-paste with name substitutions. > > Looking at chebyshev.py, it doesn't contain all that much actual code. And I can only see the extensive docstrings as a plus, not as a reason to move things over to scipy. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From zbyszek at in.waw.pl Thu Aug 26 12:06:28 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Thu, 26 Aug 2010 18:06:28 +0200 Subject: [Numpy-discussion] bug in dtype.__eq__ method, bug filed In-Reply-To: <20100825201427.GC23997@in.waw.pl> References: <20100823165009.GA5818@tulpenbaum.cognition.tu-berlin.de> <20100823165526.GB23997@in.waw.pl> <20100825201427.GC23997@in.waw.pl> Message-ID: <20100826160628.GD23997@in.waw.pl> http://projects.scipy.org/numpy/ticket/1594 From stefan at sun.ac.za Thu Aug 26 12:12:50 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 26 Aug 2010 18:12:50 +0200 Subject: [Numpy-discussion] BUG: NumPy exposes the wrong include directory In-Reply-To: References: Message-ID: On 26 August 2010 16:42, Ralf Gommers wrote: >> > Without this patch, numpy.distutils is broken on, amongs others, >> > Ubuntu. ?This prohibits packages such as Enthought's Enable from being >> > built successfully. >> > >> > Would anyone object if I applied it to trunk? >> > Ehm, didn't you apply it months ago in r8437? It hasn't broken anything, so > no objections here. Ah, so I did :-) That'll teach me to "git svn rebase" on every machine! Anyway, it's probably good that David has a look at it, just in case. Thanks for the heads up, Ralf. St?fan From bsouthey at gmail.com Thu Aug 26 12:34:20 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 26 Aug 2010 11:34:20 -0500 Subject: [Numpy-discussion] [SciPy-User] ANN: NumPy 1.5.0 release candidate 1 In-Reply-To: References: Message-ID: On Thu, Aug 26, 2010 at 6:30 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the first release candidate of > NumPy 1.5.0. This will be the first NumPy release to include support for > Python 3, as well as for Python 2.7. > > Please try this RC and report any problems on the NumPy mailing list. > Especially with Python 3 testing will be very useful. On Linux and OS X > building from source should be straightforward, for Windows a binary > installer is provided. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > > Enjoy, > Ralf > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > On my 64-bit Linux all tests pass for: Python versions: 2.4, 2.5, 2.6, 2.7, 3.1 and 3.2alpha However, I did look at the known failures by setting verbose=10. The output indicates two issues: 1) This ticket should be fixed or message changed to some future version: Ticket #1081: weird array with strange input... ... KNOWNFAIL: Fix this for 1.5.0. 2) The other is one regarding Python 3. Ticket #99 ... KNOWNFAIL: numpy.intp('0xff', 16) not supported on Py3, as it does not inherit from Python int About all I understand is that 'intp' is 'an integer that is the size of a pointer on the platform'. Anyhow, I think that at the very leaintp must have the same behavior across Python versions. With regards to Python 3, is there already an ticket for this or should this ticket be reopened? Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:31) [GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.intp('0xb72a7008', 16) 3073011720 >>> np.int('0xb72a7008', 16) 3073011720 Python 3.1.2 (r312:79147, Mar 24 2010, 10:44:23) [GCC 4.4.3 20100127 (Red Hat 4.4.3-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> int('0xb72a7008', 16) 3073011720 >>> np.intp('0xb72a7008', 16) Traceback (most recent call last): File "", line 1, in TypeError: function takes at most 1 argument (2 given) >>> np.int('0xb72a7008', 16) 3073011720 Bruce From seb.haase at gmail.com Thu Aug 26 23:43:07 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Fri, 27 Aug 2010 05:43:07 +0200 Subject: [Numpy-discussion] questions about a "complicated" user-defined dtype and the ufunc API In-Reply-To: References: Message-ID: On Thu, Aug 26, 2010 at 5:09 PM, Nathaniel Smith wrote: > On Mon, Aug 23, 2010 at 9:20 AM, Travis Oliphant wrote: >> On Aug 22, 2010, at 4:36 PM, Nathaniel Smith wrote: >>> Question 2: Am I missing something, or does the ufunc API make this >>> impossible? The problem is that a "PyUFuncGenericFunction" doesn't >>> have any way to find the dtypes of the arrays that it's working on. >>> All of the PyArray_ArrFuncs functions take a pointer to the underlying >>> ndarray as an argument, so that when working with a string or void >>> array, you can find the actual dtype and figure out (e.g.) the size of >>> the objects involved. But ufunc inner loops don't get that, so I guess >>> it's just impossible to define a ufunc over variable-sized data, or >>> data that you need to be able to see the dtype metadata to interpret? >> >> Yes, currently that is correct. ? Variable data-types don't work in ufuncs for some subtle reasons. ?But, the changes that allow date-times to work also fix (or will fix) this problem as a side-effect. >> >> The necessary changes to ufuncs have not been made, yet, however. ? They are planned. ? And, yes, this would allow ufuncs to be used for string equality testing, etc. > > Okay, thanks. It sounds like I'll need to accomplish what I want in > some other way for now, then, but if I get some surplus time somehow > then maybe I'll take a crack at the "proper" solution :-). > > -- Nathaniel Hi, Is this discussion at all related to ticket #450 ( http://projects.scipy.org/numpy/ticket/450 ) ? (( a[1:,1:,1:].min() is creating temporary copy of (potentially large) array a, if non-contiguous )) IOW, can methods be ufunc, and hence take care of memory layout complications ? -S. From faltet at pytables.org Fri Aug 27 11:22:53 2010 From: faltet at pytables.org (Francesc Alted) Date: Fri, 27 Aug 2010 17:22:53 +0200 Subject: [Numpy-discussion] [ANN] carray 0.2: an in-memory compressed data container Message-ID: ===================== Announcing carray 0.2 ===================== What it is ========== carray is a container for numerical data that can be compressed in-memory. The compresion process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. Having data compressed in-memory can reduce the stress of the memory subsystem. The net result is that carray operations can be faster than using a traditional ndarray object from NumPy. What's new ========== Two new `__iter__()` and `iter(start, stop, step)` iterators that allows to perform potentially complex operations much faster than using plain ndarrays. For example:: In [3]: a = np.arange(1e6) In [4]: time sum((v for v in a if v < 4)) CPU times: user 6.51 s, sys: 0.00 s, total: 6.51 s Wall time: 6.52 s Out[5]: 6.0 In [6]: b = ca.carray(a) In [7]: time sum((v for v in b if v < 4)) CPU times: user 0.73 s, sys: 0.04 s, total: 0.78 s Wall time: 0.75 s # 8.7x faster than ndarray Out[8]: 6.0 The `iter(start, stop, step)` iterator also allows to select slices specified by the `start`, `stop` and `step` parameters. Example:: In [9]: time sum((v for v in a[2::3] if v < 10)) CPU times: user 2.18 s, sys: 0.00 s, total: 2.18 s Wall time: 2.19 s Out[10]: 15.0 In [11]: time sum((v for v in b.iter(start=2, step=3) if v < 10)) CPU times: user 0.26 s, sys: 0.03 s, total: 0.30 s Wall time: 0.30 s # 7.3x faster than ndarray Out[12]: 15.0 The main advantage of these iterators is that you can use them in generators and hence, you don't need to waste memory for creating temporaries, which can be important when dealing with large arrays. For more info, see the new ``Using iterators`` section in USAGE.txt. Resources ========= Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://github.com/FrancescAlted/carray/downloads Short tutorial: http://github.com/FrancescAlted/carray/blob/master/USAGE.txt Home of Blosc compressor: http://blosc.pytables.org Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- Enjoy! -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett.olsen at gmail.com Fri Aug 27 15:58:45 2010 From: brett.olsen at gmail.com (Brett Olsen) Date: Fri, 27 Aug 2010 14:58:45 -0500 Subject: [Numpy-discussion] Boolean arrays Message-ID: Hello, I have an array of non-numeric data, and I want to create a boolean array denoting whether each element in this array is a "valid" value or not. This is straightforward if there's only one possible valid value: >>> import numpy as N >>> ar = N.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>> ar == "a" array([ True, False, False, False, False, True, False, False, True], dtype=bool) If there's multiple possible valid values, I've come up with a couple possible methods, but they all seem to be inefficient or kludges: >>> valid = N.array(("a", "c")) >>> (ar == valid[0]) | (ar == valid[1]) array([ True, False, True, False, False, True, False, True, True], dtype=bool) >>> N.array(map(lambda x: x in valid, ar)) array([ True, False, True, False, False, True, False, True, True], dtype=bool) Is there a numpy-appropriate way to do this? Thanks, Brett Olsen From kwatford+scipy at gmail.com Fri Aug 27 16:10:06 2010 From: kwatford+scipy at gmail.com (Ken Watford) Date: Fri, 27 Aug 2010 16:10:06 -0400 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On Fri, Aug 27, 2010 at 3:58 PM, Brett Olsen wrote: > Hello, > > I have an array of non-numeric data, and I want to create a boolean > array denoting whether each element in this array is a "valid" value > or not. ?This is straightforward if there's only one possible valid > value: >>>> import numpy as N >>>> ar = N.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>> ar == "a" > array([ True, False, False, False, False, ?True, False, False, ?True], > dtype=bool) > > If there's multiple possible valid values, I've come up with a couple > possible methods, but they all seem to be inefficient or kludges: >>>> valid = N.array(("a", "c")) >>>> (ar == valid[0]) | (ar == valid[1]) > array([ True, False, ?True, False, False, ?True, False, ?True, ?True], > dtype=bool) >>>> N.array(map(lambda x: x in valid, ar)) > array([ True, False, ?True, False, False, ?True, False, ?True, ?True], > dtype=bool) > > Is there a numpy-appropriate way to do this? > > Thanks, > Brett Olsen amap: Like Map, but for arrays. >>> ar = numpy.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>> valid = ('a', 'c') >>> numpy.amap(lambda x: x in valid, ar) array([ True, False, True, False, False, True, False, True, True], dtype=bool) From robert.kern at gmail.com Fri Aug 27 16:17:10 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 27 Aug 2010 15:17:10 -0500 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On Fri, Aug 27, 2010 at 15:10, Ken Watford wrote: > On Fri, Aug 27, 2010 at 3:58 PM, Brett Olsen wrote: >> Hello, >> >> I have an array of non-numeric data, and I want to create a boolean >> array denoting whether each element in this array is a "valid" value >> or not. ?This is straightforward if there's only one possible valid >> value: >>>>> import numpy as N >>>>> ar = N.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>>> ar == "a" >> array([ True, False, False, False, False, ?True, False, False, ?True], >> dtype=bool) >> >> If there's multiple possible valid values, I've come up with a couple >> possible methods, but they all seem to be inefficient or kludges: >>>>> valid = N.array(("a", "c")) >>>>> (ar == valid[0]) | (ar == valid[1]) >> array([ True, False, ?True, False, False, ?True, False, ?True, ?True], >> dtype=bool) >>>>> N.array(map(lambda x: x in valid, ar)) >> array([ True, False, ?True, False, False, ?True, False, ?True, ?True], >> dtype=bool) >> >> Is there a numpy-appropriate way to do this? >> >> Thanks, >> Brett Olsen > > amap: Like Map, but for arrays. > >>>> ar = numpy.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>> valid = ('a', 'c') >>>> numpy.amap(lambda x: x in valid, ar) > array([ True, False, ?True, False, False, ?True, False, ?True, ?True], > dtype=bool) I'm not sure what version of numpy this would be in; I've never seen it. But in any case, that would be very slow for large arrays since it would invoke a Python function call for every value in ar. Instead, iterate over the valid array, which is much shorter: mask = np.zeros(ar.shape, dtype=bool) for good in valid: mask |= (ar == good) Wrap that up into a function and you're good to go. That's about as efficient as it gets unless if the valid array gets large. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From njs at pobox.com Fri Aug 27 16:21:44 2010 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 27 Aug 2010 13:21:44 -0700 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On Fri, Aug 27, 2010 at 1:17 PM, Robert Kern wrote: > But in any case, that would be very slow for large arrays since it > would invoke a Python function call for every value in ar. Instead, > iterate over the valid array, which is much shorter: > > mask = np.zeros(ar.shape, dtype=bool) > for good in valid: > ? ?mask |= (ar == good) > > Wrap that up into a function and you're good to go. That's about as > efficient as it gets unless if the valid array gets large. Probably even more efficient if 'ar' is large and 'valid' is small, and shorter to boot: np.in1d(ar, valid) -- Nathaniel From aarchiba at physics.mcgill.ca Fri Aug 27 16:24:05 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Fri, 27 Aug 2010 16:24:05 -0400 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On 27 August 2010 16:17, Robert Kern wrote: > On Fri, Aug 27, 2010 at 15:10, Ken Watford wrote: >> On Fri, Aug 27, 2010 at 3:58 PM, Brett Olsen wrote: >>> Hello, >>> >>> I have an array of non-numeric data, and I want to create a boolean >>> array denoting whether each element in this array is a "valid" value >>> or not. This is straightforward if there's only one possible valid >>> value: >>>>>> import numpy as N >>>>>> ar = N.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>>>> ar == "a" >>> array([ True, False, False, False, False, True, False, False, True], >>> dtype=bool) >>> >>> If there's multiple possible valid values, I've come up with a couple >>> possible methods, but they all seem to be inefficient or kludges: >>>>>> valid = N.array(("a", "c")) >>>>>> (ar == valid[0]) | (ar == valid[1]) >>> array([ True, False, True, False, False, True, False, True, True], >>> dtype=bool) >>>>>> N.array(map(lambda x: x in valid, ar)) >>> array([ True, False, True, False, False, True, False, True, True], >>> dtype=bool) >>> >>> Is there a numpy-appropriate way to do this? >>> >>> Thanks, >>> Brett Olsen >> >> amap: Like Map, but for arrays. >> >>>>> ar = numpy.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>>> valid = ('a', 'c') >>>>> numpy.amap(lambda x: x in valid, ar) >> array([ True, False, True, False, False, True, False, True, True], >> dtype=bool) > > I'm not sure what version of numpy this would be in; I've never seen it. > > But in any case, that would be very slow for large arrays since it > would invoke a Python function call for every value in ar. Instead, > iterate over the valid array, which is much shorter: > > mask = np.zeros(ar.shape, dtype=bool) > for good in valid: > mask |= (ar == good) > > Wrap that up into a function and you're good to go. That's about as > efficient as it gets unless if the valid array gets large. The problem here is really one of how you specify which values are valid. If your only specification is with a python function, then you're stuck calling that python function once for each possible value, no way around it. But it could happen that you have an array of possible values and a corresponding boolean array that says whether they're valid or not. Then there's a shortcut that's probably faster than oring as Robert suggests: In [3]: A = np.array([1,2,6,4,4,2,1,7,8,2,2,1]) In [4]: B = np.unique1d(A) In [5]: B Out[5]: array([1, 2, 4, 6, 7, 8]) Here C specifies which ones are valid. C could be computed using some sort of validity function (which it may be possible to vectorize). In any case it's only the distinct values, and they're sorted (so you can use ranges). In [6]: C = np.array([True,True,True,False,False,True]) Now to compute validity of A: In [10]: C[np.searchsorted(B,A)] Out[10]: array([ True, True, False, True, True, True, True, False, True, True, True, True], dtype=bool) Anne > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwatford+scipy at gmail.com Fri Aug 27 16:32:14 2010 From: kwatford+scipy at gmail.com (Ken Watford) Date: Fri, 27 Aug 2010 16:32:14 -0400 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On Fri, Aug 27, 2010 at 4:17 PM, Robert Kern wrote: > On Fri, Aug 27, 2010 at 15:10, Ken Watford wrote: >> On Fri, Aug 27, 2010 at 3:58 PM, Brett Olsen wrote: >>> Hello, >>> >>> I have an array of non-numeric data, and I want to create a boolean >>> array denoting whether each element in this array is a "valid" value >>> or not. ?This is straightforward if there's only one possible valid >>> value: >>>>>> import numpy as N >>>>>> ar = N.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>>>> ar == "a" >>> array([ True, False, False, False, False, ?True, False, False, ?True], >>> dtype=bool) >>> >>> If there's multiple possible valid values, I've come up with a couple >>> possible methods, but they all seem to be inefficient or kludges: >>>>>> valid = N.array(("a", "c")) >>>>>> (ar == valid[0]) | (ar == valid[1]) >>> array([ True, False, ?True, False, False, ?True, False, ?True, ?True], >>> dtype=bool) >>>>>> N.array(map(lambda x: x in valid, ar)) >>> array([ True, False, ?True, False, False, ?True, False, ?True, ?True], >>> dtype=bool) >>> >>> Is there a numpy-appropriate way to do this? >>> >>> Thanks, >>> Brett Olsen >> >> amap: Like Map, but for arrays. >> >>>>> ar = numpy.array(("a", "b", "c", "b", "b", "a", "d", "c", "a")) >>>>> valid = ('a', 'c') >>>>> numpy.amap(lambda x: x in valid, ar) >> array([ True, False, ?True, False, False, ?True, False, ?True, ?True], >> dtype=bool) > > I'm not sure what version of numpy this would be in; I've never seen it. Ah, my fault. I started ipython in pylab mode, expected to find something like "amap", found it, and assumed it was in numpy. It's actually in matplotlib.mlab, strangely enough. From david.huard at gmail.com Fri Aug 27 16:32:48 2010 From: david.huard at gmail.com (David Huard) Date: Fri, 27 Aug 2010 16:32:48 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C7666.80703@amolf.nl> Message-ID: Nils and Joseph, Thanks for the bug report, this is now fixed in SVN (r8672). Ralph. is this something that you want to see backported in 1.5 ? Regards, David On Fri, Aug 6, 2010 at 7:49 PM, wrote: > On Fri, Aug 6, 2010 at 4:53 PM, Nils Becker wrote: > > Hi again, > > > > first a correction: I posted > > > >> I believe np.histogram(data, bins, normed=True) effectively does : > >>>> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). > >>>> > >>>> However, it _should_ do > >>>> np.histogram(data, bins, normed=False) / bins_widths > > > > but there is a normalization missing; it should read > > > > I believe np.histogram(data, bins, normed=True) effectively does > > np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]) / > data.sum() > > > > However, it _should_ do > > np.histogram(data, bins, normed=False) / bins_widths / data.sum() > > > > Bruce Southey replied: > >> As I recall, there as issues with this aspect. > >> Please search the discussion regarding histogram especially David > >> Huard's reply in this thread: > >> http://thread.gmane.org/gmane.comp.python.numeric.general/22445 > > I think this discussion pertains to a switch in calling conventions > > which happened at the time. The last reply of D. Huard (to me) seems to > > say that they did not fix anything in the _old_ semantics, but that the > > new semantics is expected to work properly. > > > > I tried with an infinite bin: > > counts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf]) > > counts > > array([1,3]) > > ncounts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf], normed=1) > > ncounts > > array([0.,0.]) > > > > this also does not make a lot of sense to me. A better result would be > > array([0.25, 0.]), since 25% of the points fall in the first bin; 75% > > fall in the second but are spread out over an infinite interval, giving > > 0. This is what my second proposal would give. I cannot find anything > > wrong with it so far... > > I didn't find any different information about the meaning of > normed=True on the mailing list nor in the trac history > > 169 > 170 if normed: > 171 db = array(np.diff(bins), float) > 172 return n/(n*db).sum(), bins > > this does not look like the correct piecewise density with unequal > binsizes. > > Thanks Nils for pointing this out, I tried only equal binsizes for a > histogram distribution. > > Josef > > > > > > > > > Cheers, Nils > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Aug 27 16:35:07 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 27 Aug 2010 15:35:07 -0500 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On Fri, Aug 27, 2010 at 15:21, Nathaniel Smith wrote: > On Fri, Aug 27, 2010 at 1:17 PM, Robert Kern wrote: >> But in any case, that would be very slow for large arrays since it >> would invoke a Python function call for every value in ar. Instead, >> iterate over the valid array, which is much shorter: >> >> mask = np.zeros(ar.shape, dtype=bool) >> for good in valid: >> ? ?mask |= (ar == good) >> >> Wrap that up into a function and you're good to go. That's about as >> efficient as it gets unless if the valid array gets large. > > Probably even more efficient if 'ar' is large and 'valid' is small, > and shorter to boot: > > np.in1d(ar, valid) Not according to my timings: [~] |2> def kern_in(x, valid): ..> mask = np.zeros(x.shape, dtype=bool) ..> for good in valid: ..> mask |= (x == good) ..> return mask ..> [~] |6> ar = np.random.randint(100, size=1000000) [~] |7> valid = np.arange(0, 100, 5) [~] |8> %timeit kern_in(ar, valid) 10 loops, best of 3: 115 ms per loop [~] |9> %timeit np.in1d(ar, valid) 1 loops, best of 3: 279 ms per loop As valid gets larger, in1d() will catch up but for smallish sizes of valid, which I suspect given the "non-numeric" nature of the OP's (Hi, Brett!) request, kern_in() is usually better. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Fri Aug 27 16:47:05 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 27 Aug 2010 15:47:05 -0500 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C7666.80703@amolf.nl> Message-ID: On Fri, Aug 27, 2010 at 15:32, David Huard wrote: > Nils and Joseph, > Thanks for the bug report, this is now fixed in SVN (r8672). While we're at it, can we change the name of the argument? "normed" has caused so much confusion over the years. We could deprecate normed=True in favor of pdf=True or density=True. We may even want to consider leaving the normed=True implementation alone with just the deprecation warning. While the behavior is incorrect, it is also very long-standing and something that people might simply have coded around. Changing the behavior without deprecation might break their workarounds silently. I admit it's a bit of a stretch, but conservativeness and coupled with the opportunity to make a desirable name change make this more attractive. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From dwf at cs.toronto.edu Fri Aug 27 17:13:03 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 27 Aug 2010 17:13:03 -0400 Subject: [Numpy-discussion] bincount() optional length argument Message-ID: <4F1925A7-CC4E-434A-B86E-4C1882E01B45@cs.toronto.edu> I've been using bincount() in situations where I always want the count vector to be a certain fixed length, even if the upper bins are 0. e.g. I want the counts of 0 through 9, and if there are no 9's in the vector I'm counting, I'd still like it to be at least length 10. It's kind of a pain to have to check for this case and concatenate 0's, nevermind an extra array allocation. How would people feel about an optional argument to support this behaviour? I'd be happy with either a "minlength" or an "exactly this length" with values above this range being ignored, but it seems like the latter might be useful in more cases. What are other people's thoughts on this? bincount is in C so it'll be a slight pain to do, however I'd be willing to do it if the patch has a reasonable chance of being accepted. David From robert.kern at gmail.com Fri Aug 27 17:15:12 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 27 Aug 2010 16:15:12 -0500 Subject: [Numpy-discussion] bincount() optional length argument In-Reply-To: <4F1925A7-CC4E-434A-B86E-4C1882E01B45@cs.toronto.edu> References: <4F1925A7-CC4E-434A-B86E-4C1882E01B45@cs.toronto.edu> Message-ID: On Fri, Aug 27, 2010 at 16:13, David Warde-Farley wrote: > I've been using bincount() in situations where I always want the count vector to be a certain fixed length, even if the upper bins are 0. e.g. I want the counts of 0 through 9, and if there are no 9's in the vector I'm counting, I'd still like it to be at least length 10. It's kind of a pain to have to check for this case and concatenate 0's, nevermind an extra array allocation. > > How would people feel about an optional argument to support this behaviour? I'd be happy with either a "minlength" or an "exactly this length" with values above this range being ignored, but it seems like the latter might be useful in more cases. minlength works for me. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From njs at pobox.com Fri Aug 27 17:55:33 2010 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 27 Aug 2010 14:55:33 -0700 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: On Fri, Aug 27, 2010 at 1:35 PM, Robert Kern wrote: > [~] > |8> %timeit kern_in(ar, valid) > 10 loops, best of 3: 115 ms per loop > > [~] > |9> %timeit np.in1d(ar, valid) > 1 loops, best of 3: 279 ms per loop > > As valid gets larger, in1d() will catch up but for smallish sizes of > valid, which I suspect given the "non-numeric" nature of the OP's (Hi, > Brett!) request, kern_in() is usually better. Oh well, I was just guessing based on algorithmic properties. Sounds like there might be some optimizations possible to in1d then, if anyone had a reason to care :-). -- Nathaniel From dwf at cs.toronto.edu Fri Aug 27 19:56:36 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 27 Aug 2010 19:56:36 -0400 Subject: [Numpy-discussion] bincount() optional length argument In-Reply-To: References: <4F1925A7-CC4E-434A-B86E-4C1882E01B45@cs.toronto.edu> Message-ID: <6A04FB9B-9A98-4778-BDE2-6847A054A53B@cs.toronto.edu> On 2010-08-27, at 5:15 PM, Robert Kern wrote: >> How would people feel about an optional argument to support this behaviour? I'd be happy with either a "minlength" or an "exactly this length" with values above this range being ignored, but it seems like the latter might be useful in more cases. > > minlength works for me. I can live with that. And hey look! It's done. http://projects.scipy.org/numpy/ticket/1595 David From charlesr.harris at gmail.com Fri Aug 27 20:43:26 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 27 Aug 2010 18:43:26 -0600 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C7666.80703@amolf.nl> Message-ID: On Fri, Aug 27, 2010 at 2:47 PM, Robert Kern wrote: > On Fri, Aug 27, 2010 at 15:32, David Huard wrote: > > Nils and Joseph, > > Thanks for the bug report, this is now fixed in SVN (r8672). > > While we're at it, can we change the name of the argument? "normed" > has caused so much confusion over the years. We could deprecate > normed=True in favor of pdf=True or density=True. > > We may even want to consider leaving the normed=True implementation > alone with just the deprecation warning. While the behavior is > incorrect, it is also very long-standing and something that people > might simply have coded around. Changing the behavior without > deprecation might break their workarounds silently. I admit it's a bit > of a stretch, but conservativeness and coupled with the opportunity to > make a desirable name change make this more attractive. > > I think that's a good approach. One possibility is do have density override normed, something like if density is not None: flowers and unicorns else: same old same old Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From zbyszek at in.waw.pl Sat Aug 28 05:12:02 2010 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Sat, 28 Aug 2010 11:12:02 +0200 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C7666.80703@amolf.nl> Message-ID: <20100828091202.GE23997@in.waw.pl> Hi, On Fri, Aug 27, 2010 at 06:43:26PM -0600, Charles R Harris wrote: > On Fri, Aug 27, 2010 at 2:47 PM, Robert Kern > wrote: > > On Fri, Aug 27, 2010 at 15:32, David Huard > wrote: > > Nils and Joseph, > > Thanks for the bug report, this is now fixed in SVN (r8672). > > While we're at it, can we change the name of the argument? "normed" > has caused so much confusion over the years. We could deprecate > normed=True in favor of pdf=True or density=True. I think it might be a good moment to also include a different type of normalization: n = n / n.sum() i.e. the frequency of counts in each bin. This one is of course very simple to calculate by hand, but very common. I think it would be useful to have this normalization available too. [http://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm] At the same time, 'normed' could be changed to 'normalized', I think that this is the standard spelling. The new API could be histogram(a, bins=10, range=None, normed=False, # deprecated normalized=None, # new weights=None) and one would pass normalized='density' to get the current behaviour of normed=True, and normalized='frequency' to get the behaviour described above. Best, Zbyszek > > We may even want to consider leaving the normed=True implementation > alone with just the deprecation warning. While the behavior is > incorrect, it is also very long-standing and something that people > might simply have coded around. Changing the behavior without > deprecation might break their workarounds silently. I admit it's a bit > of a stretch, but conservativeness and coupled with the opportunity to > make a desirable name change make this more attractive. > > I think that's a good approach. One possibility is do have density > override normed, something like > > if density is not None: > flowers and unicorns > else: > same old same old > > Chuck From faltet at pytables.org Sat Aug 28 10:14:22 2010 From: faltet at pytables.org (Francesc Alted) Date: Sat, 28 Aug 2010 15:14:22 +0100 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: 2010/8/27, Robert Kern : > [~] > |2> def kern_in(x, valid): > ..> mask = np.zeros(x.shape, dtype=bool) > ..> for good in valid: > ..> mask |= (x == good) > ..> return mask > ..> > > [~] > |6> ar = np.random.randint(100, size=1000000) > > [~] > |7> valid = np.arange(0, 100, 5) > > [~] > |8> %timeit kern_in(ar, valid) > 10 loops, best of 3: 115 ms per loop > > [~] > |9> %timeit np.in1d(ar, valid) > 1 loops, best of 3: 279 ms per loop Another possibility is to use numexpr. On a machine with 2 x E5520 quad-core processors (i.e. a total of 8 physical cores and, with hyperthreading, 16 logical cores): In [1]: import numpy as np In [2]: def kern_in(x, valid): ...: mask = np.zeros(x.shape, dtype=bool) ...: for good in valid: ...: mask |= (x == good) ...: return mask ...: In [3]: ar = np.random.randint(100, size=10000000) In [4]: valid = np.arange(0, 100, 5) In [5]: timeit kern_in(ar, valid) 1 loops, best of 3: 1.21 s per loop In [6]: sexpr = "|".join([ "(ar == %d)" % v for v in valid ]) In [7]: sexpr # (ar == 0) | (ar == 1) <==> (0,1) in ar Out[7]: '(ar == 0)|(ar == 5)|(ar == 10)|(ar == 15)|(ar == 20)|(ar == 25)|(ar == 30)|(ar == 35)|(ar == 40)|(ar == 45)|(ar == 50)|(ar == 55)|(ar == 60)|(ar == 65)|(ar == 70)|(ar == 75)|(ar == 80)|(ar == 85)|(ar == 90)|(ar == 95)' In [9]: import numexpr as nx In [10]: timeit nx.evaluate(sexpr) 10 loops, best of 3: 71.9 ms per loop That's almost 17x of speed-up wrt to kern_in() function, but not all is due to the use of the full 16 threads. Using only one thread gives: In [11]: nx.set_num_threads(1) In [12]: timeit nx.evaluate(sexpr) 1 loops, best of 3: 586 ms per loop which is about 2x faster than kern_in() for this machine. It is not always possible to use numexpr, but in this case it seems to work pretty well. -- Francesc Alted From opossumnano at gmail.com Sat Aug 28 10:58:04 2010 From: opossumnano at gmail.com (Tiziano Zito) Date: Sat, 28 Aug 2010 16:58:04 +0200 (CEST) Subject: [Numpy-discussion] =?utf-8?q?=5BANN=5D_Reminder=3A_Autumn_School_?= =?utf-8?q?=22Advanced_Scientific_Programming_in_Python=22_in_Trento=2C_It?= =?utf-8?q?aly?= Message-ID: <20100828145804.8392B2494DF@mail.bccn-berlin> Reminder: Application deadline is August 31st, 2010! =================================================== Advanced Scientific Programming in Python ========================================= an Autumn School by the G-Node, the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists actually use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques with theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We'll use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. Clean language design and easy extensibility are driving Python to become a standard tool for scientific computing. Some of the most useful open source libraries for scientific computing and visualization will be presented. This school is targeted at Post-docs and PhD students from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. A basic knowledge of the Python language is assumed. Participants without prior experience with Python should work through the proposed introductory materials. Date and Location ================= October 4th?8th, 2010. Trento, Italy. Preliminary Program =================== Day 0 (Mon Oct 4) ? Software Carpentry & Advanced Python ? Documenting code and using version control ? Object-oriented programming, design patterns, and agile programming ? Exception handling, lambdas, decorators, context managers, metaclasses Day 1 (Tue Oct 5) ? Software Carpentry ? Test-driven development, unit testing & Quality Assurance ? Debugging, profiling and benchmarking techniques ? Data serialization: from pickle to databases Day 2 (Wed Oct 6) ? Scientific Tools for Python ? Advanced NumPy ? The Quest for Speed (intro): Interfacing to C ? Programming project Day 3 (Thu Oct 7) ? The Quest for Speed ? Writing parallel applications in Python ? When parallelization does not help: the starving CPUs problem ? Programming project Day 4 (Fri Oct 8) ? Practical Software Development ? Efficient programming in teams ? Programming project ? The Pac-Man Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects Applications ============ You can apply on-line at http://www.g-node.org/python-autumnschool Applications must be submitted before August 31st, 2010. Notifications of acceptance will be sent by September 4th, 2010. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate in past editions was around 30%. Prerequisites ============= You are supposed to know the basics of Python to participate in the lectures! Look on the website for a list of introductory material. Faculty ======= ? Francesc Alted, author of PyTables, Castell? de la Plana, Spain ? Pietro Berkes, Volen Center for Complex Systems, Brandeis University, USA ? Valentin Haenel, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany ? Zbigniew J?drzejewski-Szmek, Faculty of Physics, University of Warsaw, Poland ? Eilif Muller, The Blue Brain Project, Ecole Polytechnique F?d?rale de Lausanne, Switzerland ? Emanuele Olivetti, NeuroInformatics Laboratory, Fondazione Bruno Kessler and University of Trento, Italy ? Rike-Benjamin Schuppner, Bernstein Center for Computational Neuroscience Berlin, Germany ? Bartosz Tele?czuk, Institute for Theoretical Biology, Humboldt-Universit?t zu Berlin, Germany ? Bastian Venthur, Berlin Institute of Technology and Bernstein Focus: Neurotechnology, Germany ? St?fan van der Walt, Applied Mathematics, University of Stellenbosch, South Africa ? Tiziano Zito, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany Organized by Paolo Avesani for the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler, and by Zbigniew J?drzejewski-Szmek and Tiziano Zito for the German Neuroinformatics Node of the INCF. Website: http://www.g-node.org/python-autumnschool Contact: python-info at g-node.org From robert.kern at gmail.com Sat Aug 28 11:37:57 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 28 Aug 2010 10:37:57 -0500 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: <20100828091202.GE23997@in.waw.pl> References: <4C5C7666.80703@amolf.nl> <20100828091202.GE23997@in.waw.pl> Message-ID: On Sat, Aug 28, 2010 at 04:12, Zbyszek Szmek wrote: > Hi, > > On Fri, Aug 27, 2010 at 06:43:26PM -0600, Charles R Harris wrote: >> ? ?On Fri, Aug 27, 2010 at 2:47 PM, Robert Kern >> ? ?wrote: >> >> ? ? ?On Fri, Aug 27, 2010 at 15:32, David Huard >> ? ? ?wrote: >> ? ? ?> Nils and Joseph, >> ? ? ?> Thanks for the bug report, this is now fixed in SVN (r8672). >> >> ? ? ?While we're at it, can we change the name of the argument? "normed" >> ? ? ?has caused so much confusion over the years. We could deprecate >> ? ? ?normed=True in favor of pdf=True or density=True. > I think it might be a good moment to also include a different type of normalization: > ? ? ? n = n / n.sum() > i.e. the frequency of counts in each bin. This one is of course very simple to calculate > by hand, but very common. I think it would be useful to have this normalization > available too. [http://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm] My feeling is that this is trivial to do "by hand". I do not see a reason to add an option to histogram() to do this. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From n.becker at amolf.nl Sun Aug 29 09:21:55 2010 From: n.becker at amolf.nl (Nils Becker) Date: Sun, 29 Aug 2010 15:21:55 +0200 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: Message-ID: <4C7A5EF3.3040800@amolf.nl> > On Sat, Aug 28, 2010 at 04:12, Zbyszek Szmek wrote: >> Hi, >> >> On Fri, Aug 27, 2010 at 06:43:26PM -0600, Charles R Harris wrote: >>> ? ?On Fri, Aug 27, 2010 at 2:47 PM, Robert Kern >>> ? ?wrote: >>> >>> ? ? ?On Fri, Aug 27, 2010 at 15:32, David Huard >>> ? ? ?wrote: >>> ? ? ?> Nils and Joseph, >>> ? ? ?> Thanks for the bug report, this is now fixed in SVN (r8672). >>> >>> ? ? ?While we're at it, can we change the name of the argument? "normed" >>> ? ? ?has caused so much confusion over the years. We could deprecate >>> ? ? ?normed=True in favor of pdf=True or density=True. >> I think it might be a good moment to also include a different type of normalization: >> ? ? ? n = n / n.sum() >> i.e. the frequency of counts in each bin. This one is of course very simple to calculate >> by hand, but very common. I think it would be useful to have this normalization >> available too. [http://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm] > > My feeling is that this is trivial to do "by hand". I do not see a > reason to add an option to histogram() to do this. > Hi, +1 for not silently changing the behavior of normed=True. (I'm one of the people who have worked around it). One argument in favor of putting both normalizing styles 'frequency' and 'density' may be that the documentation will automatically become very clear. A user sees all options and there is little chance of a misunderstanding. Of course, a sentence like "If you want frequency normalization, use histogram(data, normalized=False)/sum(data)" would also make things clear, without adding the frequency option. Nils From ralf.gommers at googlemail.com Sun Aug 29 10:26:23 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 29 Aug 2010 22:26:23 +0800 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C7666.80703@amolf.nl> Message-ID: On Sat, Aug 28, 2010 at 4:32 AM, David Huard wrote: > Nils and Joseph, > > Thanks for the bug report, this is now fixed in SVN (r8672). > > Ralph. is this something that you want to see backported in 1.5 ? > >From the other replies to your mail I gather your bug fix is still going to change. If no other issues are reported I'm planning to do the final release in two days, so it's a bit late for backporting. Thanks for asking, Ralf > Regards, > > David > > > On Fri, Aug 6, 2010 at 7:49 PM, wrote: > >> On Fri, Aug 6, 2010 at 4:53 PM, Nils Becker wrote: >> > Hi again, >> > >> > first a correction: I posted >> > >> >> I believe np.histogram(data, bins, normed=True) effectively does : >> >>>> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). >> >>>> >> >>>> However, it _should_ do >> >>>> np.histogram(data, bins, normed=False) / bins_widths >> > >> > but there is a normalization missing; it should read >> > >> > I believe np.histogram(data, bins, normed=True) effectively does >> > np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]) / >> data.sum() >> > >> > However, it _should_ do >> > np.histogram(data, bins, normed=False) / bins_widths / data.sum() >> > >> > Bruce Southey replied: >> >> As I recall, there as issues with this aspect. >> >> Please search the discussion regarding histogram especially David >> >> Huard's reply in this thread: >> >> http://thread.gmane.org/gmane.comp.python.numeric.general/22445 >> > I think this discussion pertains to a switch in calling conventions >> > which happened at the time. The last reply of D. Huard (to me) seems to >> > say that they did not fix anything in the _old_ semantics, but that the >> > new semantics is expected to work properly. >> > >> > I tried with an infinite bin: >> > counts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf]) >> > counts >> > array([1,3]) >> > ncounts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf], normed=1) >> > ncounts >> > array([0.,0.]) >> > >> > this also does not make a lot of sense to me. A better result would be >> > array([0.25, 0.]), since 25% of the points fall in the first bin; 75% >> > fall in the second but are spread out over an infinite interval, giving >> > 0. This is what my second proposal would give. I cannot find anything >> > wrong with it so far... >> >> I didn't find any different information about the meaning of >> normed=True on the mailing list nor in the trac history >> >> 169 >> 170 if normed: >> 171 db = array(np.diff(bins), float) >> 172 return n/(n*db).sum(), bins >> >> this does not look like the correct piecewise density with unequal >> binsizes. >> >> Thanks Nils for pointing this out, I tried only equal binsizes for a >> histogram distribution. >> >> Josef >> >> >> >> >> >> > >> > Cheers, Nils >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Aug 29 10:54:33 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 29 Aug 2010 10:54:33 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C5C7666.80703@amolf.nl> Message-ID: On Sun, Aug 29, 2010 at 10:26 AM, Ralf Gommers wrote: > > > On Sat, Aug 28, 2010 at 4:32 AM, David Huard wrote: >> >> Nils and Joseph, >> Thanks for the bug report, this is now fixed in SVN (r8672). >> Ralph. is this something that you want to see backported in 1.5 ? > > From the other replies to your mail I gather your bug fix is still going to > change. If no other issues are reported I'm planning to do the final release > in two days, so it's a bit late for backporting. There seems to be pretty much agreement on a new keyword like density=True to have the correct behavior available. Two days before a release is really not a good time, but adding an example to the docstring how to calculate the correct density histogram will be useful. Josef > > Thanks for asking, > Ralf > > >> >> Regards, >> David >> >> On Fri, Aug 6, 2010 at 7:49 PM, wrote: >>> >>> On Fri, Aug 6, 2010 at 4:53 PM, Nils Becker wrote: >>> > Hi again, >>> > >>> > first a correction: I posted >>> > >>> >> I believe np.histogram(data, bins, normed=True) effectively does : >>> >>>> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). >>> >>>> >>> >>>> However, it _should_ do >>> >>>> np.histogram(data, bins, normed=False) / bins_widths >>> > >>> > but there is a normalization missing; it should read >>> > >>> > I believe np.histogram(data, bins, normed=True) effectively does >>> > np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]) / >>> > data.sum() >>> > >>> > However, it _should_ do >>> > np.histogram(data, bins, normed=False) / bins_widths / data.sum() >>> > >>> > Bruce Southey replied: >>> >> As I recall, there as issues with this aspect. >>> >> Please search the discussion regarding histogram especially David >>> >> Huard's reply in this thread: >>> >> http://thread.gmane.org/gmane.comp.python.numeric.general/22445 >>> > I think this discussion pertains to a switch in calling conventions >>> > which happened at the time. The last reply of D. Huard (to me) seems to >>> > say that they did not fix anything in the _old_ semantics, but that the >>> > new semantics is expected to work properly. >>> > >>> > I tried with an infinite bin: >>> > counts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf]) >>> > counts >>> > array([1,3]) >>> > ncounts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf], normed=1) >>> > ncounts >>> > array([0.,0.]) >>> > >>> > this also does not make a lot of sense to me. A better result would be >>> > array([0.25, 0.]), since 25% of the points fall in the first bin; 75% >>> > fall in the second but are spread out over an infinite interval, giving >>> > 0. This is what my second proposal would give. I cannot find anything >>> > wrong with it so far... >>> >>> I didn't find any different information about the meaning of >>> normed=True on the mailing list nor in the trac history >>> >>> ? ? ? ?169 >>> ? ? ? ?170 ? ? ? ? if normed: >>> ? ? ? ?171 ? ? ? ? ? ? db = array(np.diff(bins), float) >>> ? ? ? ?172 ? ? ? ? ? ? return n/(n*db).sum(), bins >>> >>> this does not look like the correct piecewise density with unequal >>> binsizes. >>> >>> Thanks Nils for pointing this out, I tried only equal binsizes for a >>> histogram distribution. >>> >>> Josef >>> >>> >>> >>> >>> >>> > >>> > Cheers, Nils >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From seb.haase at gmail.com Sun Aug 29 17:06:54 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 29 Aug 2010 23:06:54 +0200 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: <4C7A5EF3.3040800@amolf.nl> References: <4C7A5EF3.3040800@amolf.nl> Message-ID: On Sun, Aug 29, 2010 at 3:21 PM, Nils Becker wrote: >> On Sat, Aug 28, 2010 at 04:12, Zbyszek Szmek wrote: >>> Hi, >>> >>> On Fri, Aug 27, 2010 at 06:43:26PM -0600, Charles R Harris wrote: >>>> ? ?On Fri, Aug 27, 2010 at 2:47 PM, Robert Kern >>>> ? ?wrote: >>>> >>>> ? ? ?On Fri, Aug 27, 2010 at 15:32, David Huard >>>> ? ? ?wrote: >>>> ? ? ?> Nils and Joseph, >>>> ? ? ?> Thanks for the bug report, this is now fixed in SVN (r8672). >>>> >>>> ? ? ?While we're at it, can we change the name of the argument? "normed" >>>> ? ? ?has caused so much confusion over the years. We could deprecate >>>> ? ? ?normed=True in favor of pdf=True or density=True. >>> I think it might be a good moment to also include a different type of normalization: >>> ? ? ? n = n / n.sum() >>> i.e. the frequency of counts in each bin. This one is of course very simple to calculate >>> by hand, but very common. I think it would be useful to have this normalization >>> available too. [http://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm] >> >> My feeling is that this is trivial to do "by hand". I do not see a >> reason to add an option to histogram() to do this. >> > Hi, > > +1 for not silently changing the behavior of normed=True. (I'm one of > the people who have worked around it). > > One argument in favor of putting both normalizing styles 'frequency' and > 'density' may be that the documentation will automatically become very > clear. A user sees all options and there is little chance of a > misunderstanding. Of course, a sentence like "If you want frequency > normalization, use histogram(data, normalized=False)/sum(data)" would > also make things clear, without adding the frequency option. > I am in favor of adding an option for the density mode (not for this release I guess). I often have a long expressing in place of `data` and the one extra keyword saves lot's of typing. -Sebastian Haase From david.huard at gmail.com Mon Aug 30 09:29:48 2010 From: david.huard at gmail.com (David Huard) Date: Mon, 30 Aug 2010 09:29:48 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> Message-ID: Thanks for the feedback, As far as I understand it, the proposition is to keep histogram as it is for 1.5, then in 2.0, deprecate normed=True but keep the buggy behavior, while adding a density keyword that fixes the bug. In a later release, we could then get rid of normed. While the bug won't be present in histogramdd and histogram2d, the keyword change should be mirrored in those functions as well. I personally am not too keen on changing the keyword normed for density. I feel we are trading clarity for a few new users against additional trouble for many existing users. We could mitigate this by first documenting the change in the docstring and live with both keywords for a few years before raising a DeprecationWarning. Since this has a direct impact on matloblib's hist, I'd be keen to hears the devs on this. David On Sun, Aug 29, 2010 at 5:06 PM, Sebastian Haase wrote: > On Sun, Aug 29, 2010 at 3:21 PM, Nils Becker wrote: > >> On Sat, Aug 28, 2010 at 04:12, Zbyszek Szmek wrote: > >>> Hi, > >>> > >>> On Fri, Aug 27, 2010 at 06:43:26PM -0600, Charles R Harris wrote: > >>>> ? ?On Fri, Aug 27, 2010 at 2:47 PM, Robert Kern < > robert.kern at gmail.com> > >>>> ? ?wrote: > >>>> > >>>> ? ? ?On Fri, Aug 27, 2010 at 15:32, David Huard < > david.huard at gmail.com> > >>>> ? ? ?wrote: > >>>> ? ? ?> Nils and Joseph, > >>>> ? ? ?> Thanks for the bug report, this is now fixed in SVN (r8672). > >>>> > >>>> ? ? ?While we're at it, can we change the name of the argument? > "normed" > >>>> ? ? ?has caused so much confusion over the years. We could deprecate > >>>> ? ? ?normed=True in favor of pdf=True or density=True. > >>> I think it might be a good moment to also include a different type of > normalization: > >>> ? ? ? n = n / n.sum() > >>> i.e. the frequency of counts in each bin. This one is of course very > simple to calculate > >>> by hand, but very common. I think it would be useful to have this > normalization > >>> available too. [ > http://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm] > >> > >> My feeling is that this is trivial to do "by hand". I do not see a > >> reason to add an option to histogram() to do this. > >> > > Hi, > > > > +1 for not silently changing the behavior of normed=True. (I'm one of > > the people who have worked around it). > > > > One argument in favor of putting both normalizing styles 'frequency' and > > 'density' may be that the documentation will automatically become very > > clear. A user sees all options and there is little chance of a > > misunderstanding. Of course, a sentence like "If you want frequency > > normalization, use histogram(data, normalized=False)/sum(data)" would > > also make things clear, without adding the frequency option. > > > I am in favor of adding an option for the density mode (not for this > release I guess). > I often have a long expressing in place of `data` and the one extra > keyword saves lot's of typing. > > -Sebastian Haase > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Mon Aug 30 10:19:32 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 30 Aug 2010 09:19:32 -0500 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> Message-ID: On Mon, Aug 30, 2010 at 8:29 AM, David Huard wrote: > Thanks for the feedback, > > As far as I understand it, the proposition is to keep histogram as it is > for 1.5, then in 2.0, deprecate normed=True but keep the buggy behavior, > while adding a density keyword that fixes the bug. In a later release, we > could then get rid of normed. While the bug won't be present in histogramdd > and histogram2d, the keyword change should be mirrored in those functions as > well. > > I personally am not too keen on changing the keyword normed for density. I > feel we are trading clarity for a few new users against additional trouble > for many existing users. We could mitigate this by first documenting the > change in the docstring and live with both keywords for a few years before > raising a DeprecationWarning. > > Since this has a direct impact on matloblib's hist, I'd be keen to hears > the devs on this. > > David > > I am not a dev, but I would like to give a word of warning from matplotlib. In matplotlib, the bar/hist family of functions grew organically as the devs took on various requests to add keywords and such to modify the style and behavior of those graphing functions. It has now become an unmaintainable mess, prompting discussions on how to rip it out and replace it with a cleaner implementation. While everyone agrees that it needs to be done, we all don't want to break backwards compatibility. My personal feeling is that a function should do one thing, and do that one thing well. So, to me, that means that histogram() should return an array of counts and the bins for those counts. Anything more is merely window dressing to me. With this information, one can easily compute a cumulative distribution function, and/or normalize the result. The idea is that if there is nothing special that needs to be done within the histogram algorithm to accommodate these extra features, then they belong outside the function. My 2 cents, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From danelliottster at gmail.com Mon Aug 30 11:28:05 2010 From: danelliottster at gmail.com (Daniel Elliott) Date: Mon, 30 Aug 2010 10:28:05 -0500 Subject: [Numpy-discussion] inversion of large matrices Message-ID: Hello, I am new to Python (coming from R and Matlab/Octave). I was preparing to write my usual compute pdf of a really high dimensional (e.g. 10000 dimensions) Gaussian code in Python but I noticed that numpy had a function for computing the log determinant in these situations. Is there a function for performing the inverse or even the pdf of a multinomial normal in these situations as well? Thank you. Numpy and scipy look pretty awesome. - dan From bsouthey at gmail.com Mon Aug 30 11:39:51 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 30 Aug 2010 10:39:51 -0500 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> Message-ID: <4C7BD0C7.6090903@gmail.com> On 08/30/2010 09:19 AM, Benjamin Root wrote: > On Mon, Aug 30, 2010 at 8:29 AM, David Huard > wrote: > > Thanks for the feedback, > > As far as I understand it, the proposition is to keep histogram as > it is for 1.5, then in 2.0, deprecate normed=True but keep the > buggy behavior, while adding a density keyword that fixes the bug. > In a later release, we could then get rid of normed. While the bug > won't be present in histogramdd and histogram2d, the keyword > change should be mirrored in those functions as well. > > I personally am not too keen on changing the keyword normed for > density. I feel we are trading clarity for a few new users against > additional trouble for many existing users. We could mitigate this > by first documenting the change in the docstring and live with > both keywords for a few years before raising a DeprecationWarning. > > Since this has a direct impact on matloblib's hist, I'd be keen to > hears the devs on this. > > David > > > I am not a dev, but I would like to give a word of warning from > matplotlib. > > In matplotlib, the bar/hist family of functions grew organically as > the devs took on various requests to add keywords and such to modify > the style and behavior of those graphing functions. It has now become > an unmaintainable mess, prompting discussions on how to rip it out and > replace it with a cleaner implementation. While everyone agrees that > it needs to be done, we all don't want to break backwards compatibility. > > My personal feeling is that a function should do one thing, and do > that one thing well. So, to me, that means that histogram() should > return an array of counts and the bins for those counts. Anything > more is merely window dressing to me. With this information, one can > easily compute a cumulative distribution function, and/or normalize > the result. The idea is that if there is nothing special that needs > to be done within the histogram algorithm to accommodate these extra > features, then they belong outside the function. > > My 2 cents, > Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion +1 for Ben's approach. This is very similar to my view regarding to the contingency table class proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258) . We need to provide the core functionality that other approaches such as density estimation can use but not be limited to specific details. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Aug 30 11:50:55 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 11:50:55 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: <4C7BD0C7.6090903@gmail.com> References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Mon, Aug 30, 2010 at 11:39 AM, Bruce Southey wrote: > On 08/30/2010 09:19 AM, Benjamin Root wrote: > > On Mon, Aug 30, 2010 at 8:29 AM, David Huard wrote: >> >> Thanks for the feedback, >> As far as I understand it, the proposition is to keep histogram as it is >> for 1.5, then in 2.0, deprecate normed=True but keep the buggy behavior, >> while adding a density keyword that fixes the bug. In a later release, we >> could then get rid of normed. While the bug won't be present in histogramdd >> and histogram2d, the keyword change should be mirrored in those functions as >> well. >> I personally am not too keen on changing the keyword normed for density. I >> feel we are trading clarity for a few new users against additional trouble >> for many existing users. We could mitigate this by first documenting the >> change in the docstring and live with both keywords for a few years before >> raising a DeprecationWarning. >> Since this has a direct impact on matloblib's hist, I'd be keen to hears >> the devs on this. >> David > > I am not a dev, but I would like to give a word of warning from matplotlib. > > In matplotlib, the bar/hist family of functions grew organically as the devs > took on various requests to add keywords and such to modify the style and > behavior of those graphing functions.? It has now become an unmaintainable > mess, prompting discussions on how to rip it out and replace it with a > cleaner implementation.? While everyone agrees that it needs to be done, we > all don't want to break backwards compatibility. > > My personal feeling is that a function should do one thing, and do that one > thing well.? So, to me, that means that histogram() should return an array > of counts and the bins for those counts.? Anything more is merely window > dressing to me.? With this information, one can easily compute a cumulative > distribution function, and/or normalize the result.? The idea is that if > there is nothing special that needs to be done within the histogram > algorithm to accommodate these extra features, then they belong outside the > function. > > My 2 cents, > Ben Root > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > +1 for Ben's approach. > This is very similar to my view regarding to the contingency table class > proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258). We need > to provide the core functionality that other approaches such as density > estimation can use but not be limited to specific details. I think (a corrected) density histogram is core functionality for unequal bin lengths. The graph with raw count in the case of unequal bin sizes would be quite misleading when plotted and interpreted on the real line and not on discrete points (shaded areas instead of vertical lines). And as the origin of this thread showed, it's not trivial to figure out what the correct normalization is. So, I think, if we drop the density normalization, we just need a new function that does it. My 2c, Josef > > Bruce > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From david.huard at gmail.com Mon Aug 30 13:00:18 2010 From: david.huard at gmail.com (David Huard) Date: Mon, 30 Aug 2010 13:00:18 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: I tend to agree with Josef here, To me, bincount and digitize are the low-level functions, and histogram contains a bit more functionality since its used so often and for many use cases. My guess is that if we removed the normalization, it could annoy a lot of people and would quickly appear on the desired feature list. Just to put things in perspective, this was indeed a trivial bug that required a one line fix. It only affected use cases with non-uniform bin widths and normed=True, a combination that is probably uncommon. I believe it is a genuine bug, not just a confusing behavior, and that's why I initially thought a warning was unnecessary. In any case, I'm not sure this is really a "while we're at it" situation, that is, I think the switch from "normed" to "density" should be addressed in another context. That would allow us to include the bug fix (with a warning) in the upcoming 1.5 release. David H. On Mon, Aug 30, 2010 at 11:50 AM, wrote: > On Mon, Aug 30, 2010 at 11:39 AM, Bruce Southey > wrote: > > On 08/30/2010 09:19 AM, Benjamin Root wrote: > > > > On Mon, Aug 30, 2010 at 8:29 AM, David Huard > wrote: > >> > >> Thanks for the feedback, > >> As far as I understand it, the proposition is to keep histogram as it is > >> for 1.5, then in 2.0, deprecate normed=True but keep the buggy behavior, > >> while adding a density keyword that fixes the bug. In a later release, > we > >> could then get rid of normed. While the bug won't be present in > histogramdd > >> and histogram2d, the keyword change should be mirrored in those > functions as > >> well. > >> I personally am not too keen on changing the keyword normed for density. > I > >> feel we are trading clarity for a few new users against additional > trouble > >> for many existing users. We could mitigate this by first documenting the > >> change in the docstring and live with both keywords for a few years > before > >> raising a DeprecationWarning. > >> Since this has a direct impact on matloblib's hist, I'd be keen to hears > >> the devs on this. > >> David > > > > I am not a dev, but I would like to give a word of warning from > matplotlib. > > > > In matplotlib, the bar/hist family of functions grew organically as the > devs > > took on various requests to add keywords and such to modify the style and > > behavior of those graphing functions. It has now become an > unmaintainable > > mess, prompting discussions on how to rip it out and replace it with a > > cleaner implementation. While everyone agrees that it needs to be done, > we > > all don't want to break backwards compatibility. > > > > My personal feeling is that a function should do one thing, and do that > one > > thing well. So, to me, that means that histogram() should return an > array > > of counts and the bins for those counts. Anything more is merely window > > dressing to me. With this information, one can easily compute a > cumulative > > distribution function, and/or normalize the result. The idea is that if > > there is nothing special that needs to be done within the histogram > > algorithm to accommodate these extra features, then they belong outside > the > > function. > > > > My 2 cents, > > Ben Root > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > +1 for Ben's approach. > > This is very similar to my view regarding to the contingency table class > > proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258). We > need > > to provide the core functionality that other approaches such as density > > estimation can use but not be limited to specific details. > > I think (a corrected) density histogram is core functionality for > unequal bin lengths. > > The graph with raw count in the case of unequal bin sizes would be > quite misleading when plotted and interpreted on the real line and not > on discrete points (shaded areas instead of vertical lines). And as > the origin of this thread showed, it's not trivial to figure out what > the correct normalization is. > So, I think, if we drop the density normalization, we just need a new > function that does it. > > My 2c, > > Josef > > > > > > Bruce > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.becker at amolf.nl Mon Aug 30 13:53:23 2010 From: n.becker at amolf.nl (Nils Becker) Date: Mon, 30 Aug 2010 19:53:23 +0200 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: Message-ID: <4C7BF013.6040502@amolf.nl> > I think (a corrected) density histogram is core functionality for > unequal bin lengths. > > The graph with raw count in the case of unequal bin sizes would be > quite misleading when plotted and interpreted on the real line and not > on discrete points (shaded areas instead of vertical lines). And as > the origin of this thread showed, it's not trivial to figure out what > the correct normalization is. > So, I think, if we drop the density normalization, we just need a new > function that does it. > > My 2c, > > Josef (Not a dev, but ) I agree with Josef. Throwing out normalization completely would in addition have to destroy backwards compatibility. It's another question whether the frequency ( not density! ) normalization is so simple that it is better left out, since everyone can be expected to be able to do a raw histogram and divide by the number of data points without a headache. Nils From ben.root at ou.edu Mon Aug 30 14:43:07 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 30 Aug 2010 13:43:07 -0500 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Mon, Aug 30, 2010 at 10:50 AM, wrote: > On Mon, Aug 30, 2010 at 11:39 AM, Bruce Southey > wrote: > > On 08/30/2010 09:19 AM, Benjamin Root wrote: > > > > On Mon, Aug 30, 2010 at 8:29 AM, David Huard > wrote: > >> > >> Thanks for the feedback, > >> As far as I understand it, the proposition is to keep histogram as it is > >> for 1.5, then in 2.0, deprecate normed=True but keep the buggy behavior, > >> while adding a density keyword that fixes the bug. In a later release, > we > >> could then get rid of normed. While the bug won't be present in > histogramdd > >> and histogram2d, the keyword change should be mirrored in those > functions as > >> well. > >> I personally am not too keen on changing the keyword normed for density. > I > >> feel we are trading clarity for a few new users against additional > trouble > >> for many existing users. We could mitigate this by first documenting the > >> change in the docstring and live with both keywords for a few years > before > >> raising a DeprecationWarning. > >> Since this has a direct impact on matloblib's hist, I'd be keen to hears > >> the devs on this. > >> David > > > > I am not a dev, but I would like to give a word of warning from > matplotlib. > > > > In matplotlib, the bar/hist family of functions grew organically as the > devs > > took on various requests to add keywords and such to modify the style and > > behavior of those graphing functions. It has now become an > unmaintainable > > mess, prompting discussions on how to rip it out and replace it with a > > cleaner implementation. While everyone agrees that it needs to be done, > we > > all don't want to break backwards compatibility. > > > > My personal feeling is that a function should do one thing, and do that > one > > thing well. So, to me, that means that histogram() should return an > array > > of counts and the bins for those counts. Anything more is merely window > > dressing to me. With this information, one can easily compute a > cumulative > > distribution function, and/or normalize the result. The idea is that if > > there is nothing special that needs to be done within the histogram > > algorithm to accommodate these extra features, then they belong outside > the > > function. > > > > My 2 cents, > > Ben Root > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > +1 for Ben's approach. > > This is very similar to my view regarding to the contingency table class > > proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258). We > need > > to provide the core functionality that other approaches such as density > > estimation can use but not be limited to specific details. > > I think (a corrected) density histogram is core functionality for > unequal bin lengths. > > The graph with raw count in the case of unequal bin sizes would be > quite misleading when plotted and interpreted on the real line and not > on discrete points (shaded areas instead of vertical lines). And as > the origin of this thread showed, it's not trivial to figure out what > the correct normalization is. > So, I think, if we drop the density normalization, we just need a new > function that does it. > > My 2c, > > Josef > > > Why not a function that takes the output of a core histogram and produces a correct density normalization? Such a function would be useful elsewhere, I imagine. Of course there is a lot of legacy issues to consider, but if we introduce such a function first with documentation in histogram() showing how to produce a normalized density, we can then keep some of the bad code for now for backwards compatibility with notes saying that some of the stuff will be deprecated. Especially point out in the docs where the current code fails to produce the correct results. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Mon Aug 30 14:59:24 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Mon, 30 Aug 2010 14:59:24 -0400 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: Message-ID: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> On 2010-08-30, at 11:28 AM, Daniel Elliott wrote: > Hello, > > I am new to Python (coming from R and Matlab/Octave). I was preparing > to write my usual compute pdf of a really high dimensional (e.g. 10000 > dimensions) Gaussian code in Python but I noticed that numpy had a > function for computing the log determinant in these situations. Yep. Keep in mind this is a fairly recent addition, in 1.5 I think, so if you ship code make sure to list this dependency. > Is there a function for performing the inverse or even the pdf of a > multinomial normal in these situations as well? There's a function for the inverse, but you almost never want to use it, especially if your goal is the multivariate normal density. A basic explanation of why is available here: http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ In the case of the multivariate normal density the covariance is assumed to be positive definite, and thus a Cholesky decomposition is appropriate. scipy.linalg.solve() (NOT numpy.linalg.solve()) with the sym_pos=True argument will do this for you. What do you mean by a "multinomial normal"? Do you mean multivariate normal? Offhand I can't remember if it exists in scipy.stats, but I know there's code for it in PyMC. David From josef.pktd at gmail.com Mon Aug 30 15:02:55 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 15:02:55 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Mon, Aug 30, 2010 at 2:43 PM, Benjamin Root wrote: > On Mon, Aug 30, 2010 at 10:50 AM, wrote: >> >> On Mon, Aug 30, 2010 at 11:39 AM, Bruce Southey >> wrote: >> > On 08/30/2010 09:19 AM, Benjamin Root wrote: >> > >> > On Mon, Aug 30, 2010 at 8:29 AM, David Huard >> > wrote: >> >> >> >> Thanks for the feedback, >> >> As far as I understand it, the proposition is to keep histogram as it >> >> is >> >> for 1.5, then in 2.0, deprecate normed=True but keep the buggy >> >> behavior, >> >> while adding a density keyword that fixes the bug. In a later release, >> >> we >> >> could then get rid of normed. While the bug won't be present in >> >> histogramdd >> >> and histogram2d, the keyword change should be mirrored in those >> >> functions as >> >> well. >> >> I personally am not too keen on changing the keyword normed for >> >> density. I >> >> feel we are trading clarity for a few new users against additional >> >> trouble >> >> for many existing users. We could mitigate this by first documenting >> >> the >> >> change in the docstring and live with both keywords for a few years >> >> before >> >> raising a DeprecationWarning. >> >> Since this has a direct impact on matloblib's hist, I'd be keen to >> >> hears >> >> the devs on this. >> >> David >> > >> > I am not a dev, but I would like to give a word of warning from >> > matplotlib. >> > >> > In matplotlib, the bar/hist family of functions grew organically as the >> > devs >> > took on various requests to add keywords and such to modify the style >> > and >> > behavior of those graphing functions.? It has now become an >> > unmaintainable >> > mess, prompting discussions on how to rip it out and replace it with a >> > cleaner implementation.? While everyone agrees that it needs to be done, >> > we >> > all don't want to break backwards compatibility. >> > >> > My personal feeling is that a function should do one thing, and do that >> > one >> > thing well.? So, to me, that means that histogram() should return an >> > array >> > of counts and the bins for those counts.? Anything more is merely window >> > dressing to me.? With this information, one can easily compute a >> > cumulative >> > distribution function, and/or normalize the result.? The idea is that if >> > there is nothing special that needs to be done within the histogram >> > algorithm to accommodate these extra features, then they belong outside >> > the >> > function. >> > >> > My 2 cents, >> > Ben Root >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > +1 for Ben's approach. >> > This is very similar to my view regarding to the contingency table class >> > proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258). We >> > need >> > to provide the core functionality that other approaches such as density >> > estimation can use but not be limited to specific details. >> >> I think (a corrected) density histogram is core functionality for >> unequal bin lengths. >> >> The graph with raw count in the case of unequal bin sizes would be >> quite misleading when plotted and interpreted on the real line and not >> on discrete points (shaded areas instead of vertical lines). And as >> the origin of this thread showed, it's not trivial to figure out what >> the correct normalization is. >> So, I think, if we drop the density normalization, we just need a new >> function that does it. >> >> My 2c, >> >> Josef >> >> > > Why not a function that takes the output of a core histogram and produces a > correct density normalization?? Such a function would be useful elsewhere, I > imagine. > > Of course there is a lot of legacy issues to consider, but if we introduce > such a function first with documentation in histogram() showing how to > produce a normalized density, we can then keep some of the bad code for now > for backwards compatibility with notes saying that some of the stuff will be > deprecated.? Especially point out in the docs where the current code fails > to produce the correct results. bugfix or redesign ? My feature request for (or target for forking) the histogram functions is to get the temporary results out, or get additional results, for example the bin-number or quantization for each observation, or some other things that I don't remember right now. With histogram functions that only do histograms, we loose a lot of calculations. This is, however, not really relevant for calculating densities since the bin edges are returned. Josef > > Ben Root > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From david.huard at gmail.com Mon Aug 30 15:44:17 2010 From: david.huard at gmail.com (David Huard) Date: Mon, 30 Aug 2010 15:44:17 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Mon, Aug 30, 2010 at 3:02 PM, wrote: > On Mon, Aug 30, 2010 at 2:43 PM, Benjamin Root wrote: > > On Mon, Aug 30, 2010 at 10:50 AM, wrote: > >> > >> On Mon, Aug 30, 2010 at 11:39 AM, Bruce Southey > >> wrote: > >> > On 08/30/2010 09:19 AM, Benjamin Root wrote: > >> > > >> > On Mon, Aug 30, 2010 at 8:29 AM, David Huard > >> > wrote: > >> >> > >> >> Thanks for the feedback, > >> >> As far as I understand it, the proposition is to keep histogram as it > >> >> is > >> >> for 1.5, then in 2.0, deprecate normed=True but keep the buggy > >> >> behavior, > >> >> while adding a density keyword that fixes the bug. In a later > release, > >> >> we > >> >> could then get rid of normed. While the bug won't be present in > >> >> histogramdd > >> >> and histogram2d, the keyword change should be mirrored in those > >> >> functions as > >> >> well. > >> >> I personally am not too keen on changing the keyword normed for > >> >> density. I > >> >> feel we are trading clarity for a few new users against additional > >> >> trouble > >> >> for many existing users. We could mitigate this by first documenting > >> >> the > >> >> change in the docstring and live with both keywords for a few years > >> >> before > >> >> raising a DeprecationWarning. > >> >> Since this has a direct impact on matloblib's hist, I'd be keen to > >> >> hears > >> >> the devs on this. > >> >> David > >> > > >> > I am not a dev, but I would like to give a word of warning from > >> > matplotlib. > >> > > >> > In matplotlib, the bar/hist family of functions grew organically as > the > >> > devs > >> > took on various requests to add keywords and such to modify the style > >> > and > >> > behavior of those graphing functions. It has now become an > >> > unmaintainable > >> > mess, prompting discussions on how to rip it out and replace it with a > >> > cleaner implementation. While everyone agrees that it needs to be > done, > >> > we > >> > all don't want to break backwards compatibility. > >> > > >> > My personal feeling is that a function should do one thing, and do > that > >> > one > >> > thing well. So, to me, that means that histogram() should return an > >> > array > >> > of counts and the bins for those counts. Anything more is merely > window > >> > dressing to me. With this information, one can easily compute a > >> > cumulative > >> > distribution function, and/or normalize the result. The idea is that > if > >> > there is nothing special that needs to be done within the histogram > >> > algorithm to accommodate these extra features, then they belong > outside > >> > the > >> > function. > >> > > >> > My 2 cents, > >> > Ben Root > >> > > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > +1 for Ben's approach. > >> > This is very similar to my view regarding to the contingency table > class > >> > proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258). We > >> > need > >> > to provide the core functionality that other approaches such as > density > >> > estimation can use but not be limited to specific details. > >> > >> I think (a corrected) density histogram is core functionality for > >> unequal bin lengths. > >> > >> The graph with raw count in the case of unequal bin sizes would be > >> quite misleading when plotted and interpreted on the real line and not > >> on discrete points (shaded areas instead of vertical lines). And as > >> the origin of this thread showed, it's not trivial to figure out what > >> the correct normalization is. > >> So, I think, if we drop the density normalization, we just need a new > >> function that does it. > >> > >> My 2c, > >> > >> Josef > >> > >> > > > > Why not a function that takes the output of a core histogram and produces > a > > correct density normalization? Such a function would be useful > elsewhere, I > > imagine. > > > > Of course there is a lot of legacy issues to consider, but if we > introduce > > such a function first with documentation in histogram() showing how to > > produce a normalized density, we can then keep some of the bad code for > now > > for backwards compatibility with notes saying that some of the stuff will > be > > deprecated. Especially point out in the docs where the current code > fails > > to produce the correct results. > > bugfix or redesign ? > > My feature request for (or target for forking) the histogram functions > is to get the temporary results out, or get additional results, for > example the bin-number or quantization for each observation, or some > other things that I don't remember right now. > > With histogram functions that only do histograms, we loose a lot of > calculations. This is, however, not really relevant for calculating > densities since the bin edges are returned. > > Not sure I'm understanding what you mean by this, but if you look at the code, you'll see that histogram is basically a big wrapper around a one-liner: np.diff(np.searchsorted(np.sort(data), bins)). Most of the code is there to make this one-liner user-friendly, improve performance or handle weights. I just added a warning alerting concerned users (r8674), so this takes care of the bug fix and Nils wish to avoid a silent change in behavior. These two changes could be included in 1.5 if Ralf feels this is worthwhile. Cheers, David H. > Josef > > > > > > Ben Root > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Aug 30 16:30:26 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Aug 2010 16:30:26 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Mon, Aug 30, 2010 at 3:44 PM, David Huard wrote: > > > On Mon, Aug 30, 2010 at 3:02 PM, wrote: >> >> On Mon, Aug 30, 2010 at 2:43 PM, Benjamin Root wrote: >> > On Mon, Aug 30, 2010 at 10:50 AM, wrote: >> >> >> >> On Mon, Aug 30, 2010 at 11:39 AM, Bruce Southey >> >> wrote: >> >> > On 08/30/2010 09:19 AM, Benjamin Root wrote: >> >> > >> >> > On Mon, Aug 30, 2010 at 8:29 AM, David Huard >> >> > wrote: >> >> >> >> >> >> Thanks for the feedback, >> >> >> As far as I understand it, the proposition is to keep histogram as >> >> >> it >> >> >> is >> >> >> for 1.5, then in 2.0, deprecate normed=True but keep the buggy >> >> >> behavior, >> >> >> while adding a density keyword that fixes the bug. In a later >> >> >> release, >> >> >> we >> >> >> could then get rid of normed. While the bug won't be present in >> >> >> histogramdd >> >> >> and histogram2d, the keyword change should be mirrored in those >> >> >> functions as >> >> >> well. >> >> >> I personally am not too keen on changing the keyword normed for >> >> >> density. I >> >> >> feel we are trading clarity for a few new users against additional >> >> >> trouble >> >> >> for many existing users. We could mitigate this by first documenting >> >> >> the >> >> >> change in the docstring and live with both keywords for a few years >> >> >> before >> >> >> raising a DeprecationWarning. >> >> >> Since this has a direct impact on matloblib's hist, I'd be keen to >> >> >> hears >> >> >> the devs on this. >> >> >> David >> >> > >> >> > I am not a dev, but I would like to give a word of warning from >> >> > matplotlib. >> >> > >> >> > In matplotlib, the bar/hist family of functions grew organically as >> >> > the >> >> > devs >> >> > took on various requests to add keywords and such to modify the style >> >> > and >> >> > behavior of those graphing functions.? It has now become an >> >> > unmaintainable >> >> > mess, prompting discussions on how to rip it out and replace it with >> >> > a >> >> > cleaner implementation.? While everyone agrees that it needs to be >> >> > done, >> >> > we >> >> > all don't want to break backwards compatibility. >> >> > >> >> > My personal feeling is that a function should do one thing, and do >> >> > that >> >> > one >> >> > thing well.? So, to me, that means that histogram() should return an >> >> > array >> >> > of counts and the bins for those counts.? Anything more is merely >> >> > window >> >> > dressing to me.? With this information, one can easily compute a >> >> > cumulative >> >> > distribution function, and/or normalize the result.? The idea is that >> >> > if >> >> > there is nothing special that needs to be done within the histogram >> >> > algorithm to accommodate these extra features, then they belong >> >> > outside >> >> > the >> >> > function. >> >> > >> >> > My 2 cents, >> >> > Ben Root >> >> > >> >> > _______________________________________________ >> >> > NumPy-Discussion mailing list >> >> > NumPy-Discussion at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > +1 for Ben's approach. >> >> > This is very similar to my view regarding to the contingency table >> >> > class >> >> > proposed for scipy ( http://projects.scipy.org/scipy/ticket/1258). We >> >> > need >> >> > to provide the core functionality that other approaches such as >> >> > density >> >> > estimation can use but not be limited to specific details. >> >> >> >> I think (a corrected) density histogram is core functionality for >> >> unequal bin lengths. >> >> >> >> The graph with raw count in the case of unequal bin sizes would be >> >> quite misleading when plotted and interpreted on the real line and not >> >> on discrete points (shaded areas instead of vertical lines). And as >> >> the origin of this thread showed, it's not trivial to figure out what >> >> the correct normalization is. >> >> So, I think, if we drop the density normalization, we just need a new >> >> function that does it. >> >> >> >> My 2c, >> >> >> >> Josef >> >> >> >> >> > >> > Why not a function that takes the output of a core histogram and >> > produces a >> > correct density normalization?? Such a function would be useful >> > elsewhere, I >> > imagine. >> > >> > Of course there is a lot of legacy issues to consider, but if we >> > introduce >> > such a function first with documentation in histogram() showing how to >> > produce a normalized density, we can then keep some of the bad code for >> > now >> > for backwards compatibility with notes saying that some of the stuff >> > will be >> > deprecated.? Especially point out in the docs where the current code >> > fails >> > to produce the correct results. >> >> bugfix or redesign ? >> >> My feature request for (or target for forking) the histogram functions >> is to get the temporary results out, or get additional results, for >> example the bin-number or quantization for each observation, or some >> other things that I don't remember right now. >> >> With histogram functions that only do histograms, we loose a lot of >> calculations. This is, however, not really relevant for calculating >> densities since the bin edges are returned. >> > > Not sure I'm understanding what you mean by this, but if you look at the > code, you'll see that histogram is basically a big wrapper around a > one-liner: np.diff(np.searchsorted(np.sort(data), bins)). Most of the code > is there to make this one-liner user-friendly, improve performance or handle > weights. Maybe it only applies to histogramdd. I tried to take it apart to see how it works after a discussion on the numpy mailing list, "2d binning and linear regression" on June 20th. I haven't looked at 1D histogram in a while, and it's easy to get a (slower) replacement for it. From a quick look at histogram with weights I cannot figure out if it's possible to recover the bin assignment of an observation as a byproduct Josef > I just added a warning alerting concerned users (r8674), so this takes care > of the bug fix and Nils wish to avoid a silent change in behavior. These two > changes could be included in 1.5 if Ralf feels this is worthwhile. > Cheers, > David H. > >> >> Josef >> >> >> > >> > Ben Root >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From pav at iki.fi Mon Aug 30 17:08:20 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 30 Aug 2010 21:08:20 +0000 (UTC) Subject: [Numpy-discussion] Github migration? References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Mon, 23 Aug 2010 21:15:55 +0000, Pauli Virtanen wrote: [clip] > in the history to have the wrong content -- so to be sure, we have to do > a brute-force comparison of the tree against SVN for each commit. The > particular bug here was fixed in the conversion tool, but better safe > than sorry. Also this is now done. The repository at http://github.com/numpy/numpy_svn_test4 is essentially fine. You can grab it via git clone git://github.com/numpy/numpy_svn_test4.git The deleted SVN branches can also be obtained: git fetch origin refs/svn/*:refs/remotes/svn/* They are not pulled by default. *** svn-all-fast-export still has some bugs, so revisions that in SVN are r2478, r2491, r2534, r2602, r2603, r4113, r4179, r4180 contain the wrong tree in Git. The errors are - Wrong branch point for branches/arraykit (an abandoned and deleted branch that was never merged back to trunk) - The commits r4113, r4179, r4180 contain changes in wrong order (the subsequent commits are OK). I'd be willing just to ignore these. Such small errors should not matter. *** So as far as I see, we would be ready to go with this. Pauli From melissawm at gmail.com Mon Aug 30 17:42:52 2010 From: melissawm at gmail.com (=?ISO-8859-1?Q?Melissa_Mendon=E7a?=) Date: Mon, 30 Aug 2010 18:42:52 -0300 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: Hi, I've been lurking for a while here but never really introduced myself. I'm a mathematician in Brazil working with optimization and numerical analysis and I'm looking into scipy/numpy basically because I want to ditch matlab. I'm just curious as to why you say "scipy.linalg.solve(), NOT numpy.linalg.solve()". Can you explain the reason for this? I find myself looking for information such as this on the internet but I rarely find real documentation for these things, and I seem to have so many performance issues with python... I'm curious to see what I'm missing here. Thanks, and sorry if I hijacked the thread, - Melissa On Mon, Aug 30, 2010 at 3:59 PM, David Warde-Farley wrote: > > On 2010-08-30, at 11:28 AM, Daniel Elliott wrote: > >> Hello, >> >> I am new to Python (coming from R and Matlab/Octave). ?I was preparing >> to write my usual compute pdf of a really high dimensional (e.g. 10000 >> dimensions) Gaussian code in Python but I noticed that numpy had a >> function for computing the log determinant in these situations. > > Yep. Keep in mind this is a fairly recent addition, in 1.5 I think, so if you ship code make sure to list this dependency. > >> Is there a function for performing the inverse or even the pdf of a >> multinomial normal in these situations as well? > > > There's a function for the inverse, but you almost never want to use it, especially if your goal is the multivariate normal density. A basic explanation of why is available here: http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ > > In the case of the multivariate normal density the covariance is assumed to be positive definite, and thus a Cholesky decomposition is appropriate. scipy.linalg.solve() (NOT numpy.linalg.solve()) with the sym_pos=True argument will do this for you. > > What do you mean by a "multinomial normal"? Do you mean multivariate normal? Offhand I can't remember if it exists in scipy.stats, but I know there's code for it in PyMC. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Melissa Weber Mendon?a -- "Knowledge is knowing a tomato is a fruit; wisdom is knowing you don't put tomato in fruit salad." From dwf at cs.toronto.edu Mon Aug 30 19:22:16 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Mon, 30 Aug 2010 19:22:16 -0400 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: On 2010-08-30, at 5:42 PM, Melissa Mendon?a wrote: > I'm just curious as to why you say "scipy.linalg.solve(), NOT > numpy.linalg.solve()". Can you explain the reason for this? Oh, the performance will be similar, provided you've linked against a good BLAS. It's just that the NumPy version doesn't have the sym_pos keyword. As for why that is: http://ask.scipy.org/en/topic/10-what-s-the-difference-between-numpy-linalg-and-scipy-linalg David From charlesr.harris at gmail.com Mon Aug 30 21:09:35 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Aug 2010 19:09:35 -0600 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: On Mon, Aug 30, 2010 at 3:42 PM, Melissa Mendon?a wrote: > Hi, > > I've been lurking for a while here but never really introduced myself. > I'm a mathematician in Brazil working with optimization and numerical > analysis and I'm looking into scipy/numpy basically because I want to > ditch matlab. > > I'm just curious as to why you say "scipy.linalg.solve(), NOT > numpy.linalg.solve()". Can you explain the reason for this? I find > myself looking for information such as this on the internet but I > rarely find real documentation for these things, and I seem to have so > many performance issues with python... I'm curious to see what I'm > missing here. > > What sort of performance issues are you having? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From danelliottster at gmail.com Mon Aug 30 22:19:46 2010 From: danelliottster at gmail.com (Dan Elliott) Date: Tue, 31 Aug 2010 02:19:46 +0000 (UTC) Subject: [Numpy-discussion] inversion of large matrices References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: Thanks for the reply. David Warde-Farley cs.toronto.edu> writes: > On 2010-08-30, at 11:28 AM, Daniel Elliott wrote: > > Large matrices (e.g. 10K x 10K) > > > Is there a function for performing the inverse or even the pdf of a > > multinomial normal in these situations as well? > > There's a function for the inverse, but you almost never want to use it, especially if your goal is the > multivariate normal density. A basic explanation of why is available here: > http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ > > In the case of the multivariate normal density the covariance is assumed to be positive definite, and thus a > Cholesky decomposition is appropriate. scipy.linalg.solve() (NOT numpy.linalg.solve()) with the > sym_pos=True argument will do this for you. You don't think this will choke on a large (e.g. 10K x 10K) covariance matrix? Given what you know about how it computes the log determinant and how the Cholesky decomposition, do you suppose I would be better off using eigen- decomposition to do this since I will also get the determinant from the sum of the logs of the eigenvalues? > What do you mean by a "multinomial normal"? Do you mean multivariate normal? Offhand I can't remember if it > exists in scipy.stats, but I know there's code for it in PyMC. Oops, I meant multivariate normal. I found code in the pymix library and so far I like what I see there. I am sure it works well for most cases. However, with the exception of one R library, I have never found a packaged implementation that works on covariance matrices as large as what I am talking about above. From david at silveregg.co.jp Mon Aug 30 22:35:28 2010 From: david at silveregg.co.jp (David) Date: Tue, 31 Aug 2010 11:35:28 +0900 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: <4C7C6A70.9000404@silveregg.co.jp> On 08/31/2010 11:19 AM, Dan Elliott wrote: > Thanks for the reply. > > David Warde-Farley cs.toronto.edu> writes: >> On 2010-08-30, at 11:28 AM, Daniel Elliott wrote: >>> Large matrices (e.g. 10K x 10K) >> >>> Is there a function for performing the inverse or even the pdf of a >>> multinomial normal in these situations as well? >> >> There's a function for the inverse, but you almost never want to use it, > especially if your goal is the >> multivariate normal density. A basic explanation of why is available here: >> http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ >> >> In the case of the multivariate normal density the covariance is assumed to be > positive definite, and thus a >> Cholesky decomposition is appropriate. scipy.linalg.solve() (NOT > numpy.linalg.solve()) with the >> sym_pos=True argument will do this for you. > > You don't think this will choke on a large (e.g. 10K x 10K) covariance matrix? It will work if you have enough memory. I have worked with slightly bigger matrices, but I have 12 Gb on my machine. You need some patience, though :) cheers, David From charlesr.harris at gmail.com Mon Aug 30 22:36:12 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Aug 2010 20:36:12 -0600 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: On Mon, Aug 30, 2010 at 8:19 PM, Dan Elliott wrote: > Thanks for the reply. > > David Warde-Farley cs.toronto.edu> writes: > > On 2010-08-30, at 11:28 AM, Daniel Elliott wrote: > > > Large matrices (e.g. 10K x 10K) > > > > > Is there a function for performing the inverse or even the pdf of a > > > multinomial normal in these situations as well? > > > > There's a function for the inverse, but you almost never want to use it, > especially if your goal is the > > multivariate normal density. A basic explanation of why is available > here: > > http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ > > > > In the case of the multivariate normal density the covariance is assumed > to be > positive definite, and thus a > > Cholesky decomposition is appropriate. scipy.linalg.solve() (NOT > numpy.linalg.solve()) with the > > sym_pos=True argument will do this for you. > > You don't think this will choke on a large (e.g. 10K x 10K) covariance > matrix? > > Should work, give it a shot. It's an n^3 problem, so might take a bit. > Given what you know about how it computes the log determinant and how the > Cholesky decomposition, do you suppose I would be better off using eigen- > decomposition to do this since I will also get the determinant from the sum > of > the logs of the eigenvalues? > > I don't see what the connection with the determinant is. The log determinant will be calculated using the ordinary LU decomposition as that works for more general matrices. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Tue Aug 31 04:57:27 2010 From: markbak at gmail.com (Mark Bakker) Date: Tue, 31 Aug 2010 10:57:27 +0200 Subject: [Numpy-discussion] conversion to 1D array of specified dtype Message-ID: Hello list, What is the easiest way to convert a function argument to at least a 1D array of specified dtype? atleast_1d(3,dtype='d') doesn't work (numpy 1.3.0) array(atleast_1d(3),dtype='d') works but seems cumbersome Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilcrighton at gmail.com Tue Aug 31 06:00:03 2010 From: neilcrighton at gmail.com (Neil) Date: Tue, 31 Aug 2010 10:00:03 +0000 (UTC) Subject: [Numpy-discussion] Boolean arrays References: Message-ID: Nathaniel Smith pobox.com> writes: > On Fri, Aug 27, 2010 at 1:35 PM, Robert Kern gmail.com> wrote: > > As valid gets larger, in1d() will catch up but for smallish sizes of > > valid, which I suspect given the "non-numeric" nature of the OP's (Hi, > > Brett!) request, kern_in() is usually better. > > Oh well, I was just guessing based on algorithmic properties. Sounds > like there might be some optimizations possible to in1d then, if > anyone had a reason to care . > Ideally, I would like in1d to always be the right answer to this problem. It should be easy to put in an if statement to switch to a kern_in()-type function in the case of large ar1 but small ar2. I will do some timing tests and make a patch. Incidentally, the timing tests done when in1d was introduced only considered the case when len(ar1) = len(ar2). In this case the current in_1d is pretty much always faster than kern_in(). Neil From ralf.gommers at googlemail.com Tue Aug 31 07:02:03 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 31 Aug 2010 19:02:03 +0800 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Tue, Aug 31, 2010 at 3:44 AM, David Huard wrote: > > I just added a warning alerting concerned users (r8674), so this takes care > of the bug fix and Nils wish to avoid a silent change in behavior. These two > changes could be included in 1.5 if Ralf feels this is worthwhile. > > That looks like a reasonable solution. I haven't got a strong opinion on whether or not to change the 'normed' keyword to 'density'. Looking at the changes, I don't think that is the right way to do the filtering in the tests. resetwarnings() removes all filters including the ones previously set by users, and should therefore not be used. Better to either raise a specific warning and filter on that, or to filter on the message content with: warnings.filterwarnings('ignore' , message="This release of NumPy fixes a normalization bug in histogram"). I found one more place where resetwarnings() is used, in test_arraysetops.py, I'll change that in trunk. Related problem there is that the warning in warnings.warn is not a DeprecationWarning. The above problem is easy to fix, but in any case it's too late to go into 1.5.0 - I'll tag the final release tonight. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Tue Aug 31 09:20:23 2010 From: rmay31 at gmail.com (Ryan May) Date: Tue, 31 Aug 2010 08:20:23 -0500 Subject: [Numpy-discussion] conversion to 1D array of specified dtype In-Reply-To: References: Message-ID: On Tue, Aug 31, 2010 at 3:57 AM, Mark Bakker wrote: > Hello list, > > What is the easiest way to convert a function argument to at least a 1D > array of specified dtype? > > atleast_1d(3,dtype='d') doesn't work (numpy 1.3.0) > > array(atleast_1d(3),dtype='d') works but seems cumbersome atleast_1d(d).astype('d') Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From david.huard at gmail.com Tue Aug 31 10:13:13 2010 From: david.huard at gmail.com (David Huard) Date: Tue, 31 Aug 2010 10:13:13 -0400 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Tue, Aug 31, 2010 at 7:02 AM, Ralf Gommers wrote: > > > On Tue, Aug 31, 2010 at 3:44 AM, David Huard wrote: > >> >> I just added a warning alerting concerned users (r8674), so this takes >> care of the bug fix and Nils wish to avoid a silent change in behavior. >> These two changes could be included in 1.5 if Ralf feels this is >> worthwhile. >> >> That looks like a reasonable solution. I haven't got a strong opinion on > whether or not to change the 'normed' keyword to 'density'. > > Looking at the changes, I don't think that is the right way to do the > filtering in the tests. resetwarnings() removes all filters including the > ones previously set by users, and should therefore not be used. Better to > either raise a specific warning and filter on that, or to filter on the > message content with: > warnings.filterwarnings('ignore' , message="This release of NumPy fixes > a normalization bug in histogram"). > I found one more place where resetwarnings() is used, in > test_arraysetops.py, I'll change that in trunk. Related problem there is > that the warning in warnings.warn is not a DeprecationWarning. > > The above problem is easy to fix, but in any case it's too late to go into > 1.5.0 - I'll tag the final release tonight. > > Ralf, test_function_base and test_arraysetops now do not use resetwarnings. What I did is added a warning filter and popped it out of the filters list afterwards. Is this OK ? In other tests, what is done is rather warnings.simplefilter('ignore', DeprecationWarning) test_function() warnings.simplefilter('default', DeprecationWarning) but that will also override any user-defined setup, no ? David Cheers, > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Aug 31 10:27:10 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 31 Aug 2010 22:27:10 +0800 Subject: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior) In-Reply-To: References: <4C7A5EF3.3040800@amolf.nl> <4C7BD0C7.6090903@gmail.com> Message-ID: On Tue, Aug 31, 2010 at 10:13 PM, David Huard wrote: > > > On Tue, Aug 31, 2010 at 7:02 AM, Ralf Gommers > wrote: > >> >> >> On Tue, Aug 31, 2010 at 3:44 AM, David Huard wrote: >> >>> >>> I just added a warning alerting concerned users (r8674), so this takes >>> care of the bug fix and Nils wish to avoid a silent change in behavior. >>> These two changes could be included in 1.5 if Ralf feels this is >>> worthwhile. >>> >>> That looks like a reasonable solution. I haven't got a strong opinion on >> whether or not to change the 'normed' keyword to 'density'. >> >> Looking at the changes, I don't think that is the right way to do the >> filtering in the tests. resetwarnings() removes all filters including the >> ones previously set by users, and should therefore not be used. Better to >> either raise a specific warning and filter on that, or to filter on the >> message content with: >> warnings.filterwarnings('ignore' , message="This release of NumPy >> fixes a normalization bug in histogram"). >> I found one more place where resetwarnings() is used, in >> test_arraysetops.py, I'll change that in trunk. Related problem there is >> that the warning in warnings.warn is not a DeprecationWarning. >> >> The above problem is easy to fix, but in any case it's too late to go into >> 1.5.0 - I'll tag the final release tonight. >> >> > Ralf, > > test_function_base and test_arraysetops now do not use resetwarnings. What > I did is added a warning filter and popped it out of the filters list > afterwards. Is this OK ? > That looks like the best solution to me (besides the catch_warnings context manager, which we can't use). > > In other tests, what is done is rather > > warnings.simplefilter('ignore', DeprecationWarning) > test_function() > warnings.simplefilter('default', DeprecationWarning) > > but that will also override any user-defined setup, no ? > Yes, it's less invasive than resetwarnings() but could also be annoying for some users. Would be good to do this in a consistent way for all warnings and tests, and describe that in http://projects.scipy.org/numpy/wiki/ApiDeprecation. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Aug 31 12:18:23 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 1 Sep 2010 00:18:23 +0800 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 release Message-ID: I am pleased to announce the availability of NumPy 1.5.0. This is the first NumPy release to include support for Python 3, as well as for Python 2.7. Binaries, sources, documentation and release notes can be found at https://sourceforge.net/projects/numpy/files/. Thank you to everyone who contributed to this release. Enjoy, the Numpy developers ========================= NumPy 1.5.0 Release Notes ========================= Highlights ========== Python 3 compatibility ---------------------- This is the first NumPy release which is compatible with Python 3. Support for Python 3 and Python 2 is done from a single code base. Extensive notes on changes can be found at ``_. Note that the Numpy testing framework relies on nose, which does not have a Python 3 compatible release yet. A working Python 3 branch of nose can be found at ``_ however. Porting of SciPy to Python 3 is expected to be completed soon. :pep:`3118` compatibility ------------------------- The new buffer protocol described by PEP 3118 is fully supported in this version of Numpy. On Python versions >= 2.6 Numpy arrays expose the buffer interface, and array(), asarray() and other functions accept new-style buffers as input. New features ============ Warning on casting complex to real ---------------------------------- Numpy now emits a `numpy.ComplexWarning` when a complex number is cast into a real number. For example: >>> x = np.array([1,2,3]) >>> x[:2] = np.array([1+2j, 1-2j]) ComplexWarning: Casting complex values to real discards the imaginary part The cast indeed discards the imaginary part, and this may not be the intended behavior in all cases, hence the warning. This warning can be turned off in the standard way: >>> import warnings >>> warnings.simplefilter("ignore", np.ComplexWarning) Dot method for ndarrays ----------------------- Ndarrays now have the dot product also as a method, which allows writing chains of matrix products as >>> a.dot(b).dot(c) instead of the longer alternative >>> np.dot(a, np.dot(b, c)) linalg.slogdet function ----------------------- The slogdet function returns the sign and logarithm of the determinant of a matrix. Because the determinant may involve the product of many small/large values, the result is often more accurate than that obtained by simple multiplication. new header ---------- The new header file ndarraytypes.h contains the symbols from ndarrayobject.h that do not depend on the PY_ARRAY_UNIQUE_SYMBOL and NO_IMPORT/_ARRAY macros. Broadly, these symbols are types, typedefs, and enumerations; the array function calls are left in ndarrayobject.h. This allows users to include array-related types and enumerations without needing to concern themselves with the macro expansions and their side- effects. Changes ======= polynomial.polynomial --------------------- * The polyint and polyder functions now check that the specified number integrations or derivations is a non-negative integer. The number 0 is a valid value for both functions. * A degree method has been added to the Polynomial class. * A trimdeg method has been added to the Polynomial class. It operates like truncate except that the argument is the desired degree of the result, not the number of coefficients. * Polynomial.fit now uses None as the default domain for the fit. The default Polynomial domain can be specified by using [] as the domain value. * Weights can be used in both polyfit and Polynomial.fit * A linspace method has been added to the Polynomial class to ease plotting. * The polymulx function was added. polynomial.chebyshev -------------------- * The chebint and chebder functions now check that the specified number integrations or derivations is a non-negative integer. The number 0 is a valid value for both functions. * A degree method has been added to the Chebyshev class. * A trimdeg method has been added to the Chebyshev class. It operates like truncate except that the argument is the desired degree of the result, not the number of coefficients. * Chebyshev.fit now uses None as the default domain for the fit. The default Chebyshev domain can be specified by using [] as the domain value. * Weights can be used in both chebfit and Chebyshev.fit * A linspace method has been added to the Chebyshev class to ease plotting. * The chebmulx function was added. * Added functions for the Chebyshev points of the first and second kind. histogram --------- After a two years transition period, the old behavior of the histogram function has been phased out, and the "new" keyword has been removed. correlate --------- The old behavior of correlate was deprecated in 1.4.0, the new behavior (the usual definition for cross-correlation) is now the default. Checksums ========= 738572dd3e5d4cd89e98c76cc3f162a9 release/installers/numpy-1.5.0-py2.5-python.org.dmg f58ebc1840974cf2ef8b4f41516dc288 release/installers/numpy-1.5.0-py2.6-python.org.dmg d7232048392ede8d8d8fb57839cb4b91 release/installers/numpy-1.5.0-py2.7-python.org.dmg c5130a11f920706cdc665ef1e07772e2 release/installers/numpy-1.5.0-win32-superpack-python2.5.exe b46a52f82126ace1eb7cb627623c64dc release/installers/numpy-1.5.0-win32-superpack-python2.6.exe 8a93c004a104f6231de4c398e2b3c48f release/installers/numpy-1.5.0-win32-superpack-python2.7.exe 1d255b58764d465e64b7b3eee832aa9e release/installers/numpy-1.5.0-win32-superpack-python3.1.exe 3a8bfdc434df782d647161c48943ee09 release/installers/numpy-1.5.0.tar.gz 11354c0ca15ca6f37df9589bd4f25943 release/installers/numpy-1.5.0.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: From eadrogue at gmx.net Tue Aug 31 12:35:27 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Tue, 31 Aug 2010 18:35:27 +0200 Subject: [Numpy-discussion] comparison between arrays of strings and numerical types Message-ID: <20100831163527.GA14551@doriath.local> Hi, I find this a bit odd: In [18]: np.array(['a','b','c','d']) > 'a' Out[18]: array([False, True, True, True], dtype=bool) In [19]: np.array(['a','b','c','d']) > 4 Out[19]: True In [20]: np.array(['a','b','c','d']) > 4.5 Out[20]: True Is that right? I was expecting an element-wise comparison, but it returns a single truth value. Cheers, Ernest From kwgoodman at gmail.com Tue Aug 31 12:44:53 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 31 Aug 2010 09:44:53 -0700 Subject: [Numpy-discussion] comparison between arrays of strings and numerical types In-Reply-To: <20100831163527.GA14551@doriath.local> References: <20100831163527.GA14551@doriath.local> Message-ID: 2010/8/31 Ernest Adrogu? : > Hi, > > I find this a bit odd: > > In [18]: np.array(['a','b','c','d']) > 'a' > Out[18]: array([False, ?True, ?True, ?True], dtype=bool) > > In [19]: np.array(['a','b','c','d']) > 4 > Out[19]: True > > In [20]: np.array(['a','b','c','d']) > 4.5 > Out[20]: True > > Is that right? I was expecting an element-wise comparison, > but it returns a single truth value. Here's a recent discussion of the issue: http://www.mail-archive.com/numpy-discussion at scipy.org/msg27051.html From charlesr.harris at gmail.com Tue Aug 31 12:56:23 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 Aug 2010 10:56:23 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.5.0 release In-Reply-To: References: Message-ID: On Tue, Aug 31, 2010 at 10:18 AM, Ralf Gommers wrote: > I am pleased to announce the availability of NumPy 1.5.0. This is the first > NumPy release to include support for Python 3, as well as for Python 2.7. > > Binaries, sources, documentation and release notes can be found at > https://sourceforge.net/projects/numpy/files/. > > Thank you to everyone who contributed to this release. > > Enjoy, > the Numpy developers > > > ========================= > NumPy 1.5.0 Release Notes > ========================= > > > Highlights > ========== > > Python 3 compatibility > ---------------------- > > This is the first NumPy release which is compatible with Python 3. Support > for > Python 3 and Python 2 is done from a single code base. Extensive notes on > changes can be found at > ``_. > > Note that the Numpy testing framework relies on nose, which does not have a > Python 3 compatible release yet. A working Python 3 branch of nose can be > found > at ``_ however. > > Porting of SciPy to Python 3 is expected to be completed soon. > > :pep:`3118` compatibility > ------------------------- > > The new buffer protocol described by PEP 3118 is fully supported in this > version of Numpy. On Python versions >= 2.6 Numpy arrays expose the buffer > interface, and array(), asarray() and other functions accept new-style > buffers > as input. > > > New features > ============ > > Warning on casting complex to real > ---------------------------------- > > Numpy now emits a `numpy.ComplexWarning` when a complex number is cast > into a real number. For example: > > >>> x = np.array([1,2,3]) > >>> x[:2] = np.array([1+2j, 1-2j]) > ComplexWarning: Casting complex values to real discards the imaginary > part > > The cast indeed discards the imaginary part, and this may not be the > intended behavior in all cases, hence the warning. This warning can be > turned off in the standard way: > > >>> import warnings > >>> warnings.simplefilter("ignore", np.ComplexWarning) > > Dot method for ndarrays > ----------------------- > > Ndarrays now have the dot product also as a method, which allows writing > chains of matrix products as > > >>> a.dot(b).dot(c) > > instead of the longer alternative > > >>> np.dot(a, np.dot(b, c)) > > linalg.slogdet function > ----------------------- > > The slogdet function returns the sign and logarithm of the determinant > of a matrix. Because the determinant may involve the product of many > small/large values, the result is often more accurate than that obtained > by simple multiplication. > > new header > ---------- > > The new header file ndarraytypes.h contains the symbols from > ndarrayobject.h that do not depend on the PY_ARRAY_UNIQUE_SYMBOL and > NO_IMPORT/_ARRAY macros. Broadly, these symbols are types, typedefs, > and enumerations; the array function calls are left in > ndarrayobject.h. This allows users to include array-related types and > enumerations without needing to concern themselves with the macro > expansions and their side- effects. > > > Changes > ======= > > polynomial.polynomial > --------------------- > > * The polyint and polyder functions now check that the specified number > integrations or derivations is a non-negative integer. The number 0 is > a valid value for both functions. > * A degree method has been added to the Polynomial class. > * A trimdeg method has been added to the Polynomial class. It operates like > truncate except that the argument is the desired degree of the result, > not the number of coefficients. > * Polynomial.fit now uses None as the default domain for the fit. The > default > Polynomial domain can be specified by using [] as the domain value. > * Weights can be used in both polyfit and Polynomial.fit > * A linspace method has been added to the Polynomial class to ease > plotting. > * The polymulx function was added. > > polynomial.chebyshev > -------------------- > > * The chebint and chebder functions now check that the specified number > integrations or derivations is a non-negative integer. The number 0 is > a valid value for both functions. > * A degree method has been added to the Chebyshev class. > * A trimdeg method has been added to the Chebyshev class. It operates like > truncate except that the argument is the desired degree of the result, > not the number of coefficients. > * Chebyshev.fit now uses None as the default domain for the fit. The > default > Chebyshev domain can be specified by using [] as the domain value. > * Weights can be used in both chebfit and Chebyshev.fit > * A linspace method has been added to the Chebyshev class to ease plotting. > * The chebmulx function was added. > * Added functions for the Chebyshev points of the first and second kind. > > > histogram > --------- > > After a two years transition period, the old behavior of the histogram > function > has been phased out, and the "new" keyword has been removed. > > correlate > --------- > > The old behavior of correlate was deprecated in 1.4.0, the new behavior > (the > usual definition for cross-correlation) is now the default. > > Checksums > ========= > > 738572dd3e5d4cd89e98c76cc3f162a9 > release/installers/numpy-1.5.0-py2.5-python.org.dmg > f58ebc1840974cf2ef8b4f41516dc288 > release/installers/numpy-1.5.0-py2.6-python.org.dmg > d7232048392ede8d8d8fb57839cb4b91 > release/installers/numpy-1.5.0-py2.7-python.org.dmg > c5130a11f920706cdc665ef1e07772e2 > release/installers/numpy-1.5.0-win32-superpack-python2.5.exe > b46a52f82126ace1eb7cb627623c64dc > release/installers/numpy-1.5.0-win32-superpack-python2.6.exe > 8a93c004a104f6231de4c398e2b3c48f > release/installers/numpy-1.5.0-win32-superpack-python2.7.exe > 1d255b58764d465e64b7b3eee832aa9e > release/installers/numpy-1.5.0-win32-superpack-python3.1.exe > 3a8bfdc434df782d647161c48943ee09 release/installers/numpy-1.5.0.tar.gz > 11354c0ca15ca6f37df9589bd4f25943 release/installers/numpy-1.5.0.zip > > And a special thanks to you, Ralf. You have done wonderful job taking hold of the release process and keeping numpy development going down the road. On to 2.0, heigh-ho. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Aug 31 13:03:55 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 31 Aug 2010 12:03:55 -0500 Subject: [Numpy-discussion] Python3 and intp error (related to ticket 99) Message-ID: <4C7D35FB.1080804@gmail.com> Hi, I was curious why there was a difference in number of known failures between Python2.6 and Python3.1 which is associated a test due to ticket 99: http://projects.scipy.org/numpy/ticket/99 While this ticket was closed, it fails with Python 3.1 as indicated by the message of the test output is: 'Ticket #99 ... KNOWNFAIL: numpy.intp('0xff', 16) not supported on Py3, as it does not inherit from Python int' I do understand that np.intp is integer size of a pointer. But it appears to be mainly used for access to C programs. The only Python numpy usage I saw was with the delete and insert function in 'numpy/lib/function_base.py'. Does this really need to be exposed in Python? If it does not, then could be removed for Numpy 2? Otherwise, at the very least np.intp must have the same behavior across Python versions. As per the ticket, since it is an integer type, should it have the same properties as a regular integer? Bruce $ python3.1 Python 3.1.2 (r312:79147, Mar 24 2010, 10:44:23) [GCC 4.4.3 20100127 (Red Hat 4.4.3-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.5.0' >>> numpy.intp('0xff', 16) Traceback (most recent call last): File "", line 1, in TypeError: function takes at most 1 argument (2 given) >>> numpy.intp('0xff') Traceback (most recent call last): File "", line 1, in ValueError: setting an array element with a sequence. >>> Yet there is no problem with Python 2.6: $ python Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:31) [GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.5.0' >>> numpy.intp('0xff', 16) 255 >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Aug 31 13:14:46 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 31 Aug 2010 17:14:46 +0000 (UTC) Subject: [Numpy-discussion] Python3 and intp error (related to ticket 99) References: <4C7D35FB.1080804@gmail.com> Message-ID: Tue, 31 Aug 2010 12:03:55 -0500, Bruce Southey wrote: [clip] > I do understand that np.intp is integer size of a pointer. But it > appears to be mainly used for access to C programs. The only Python > numpy usage I saw was with the delete and insert function in > 'numpy/lib/function_base.py'. > > Does this really need to be exposed in Python? If it does not, then > could be removed for Numpy 2? Yes, it needs to be exposed, since it can be used in data type definitions. > Otherwise, at the very least np.intp must have the same behavior across > Python versions. As per the ticket, since it is an integer type, should > it have the same properties as a regular integer? This is not specific to intp, but applies to all Numpy integer types on Python 3, since they no longer inherit the constructor from the Python integer. I note that on Python 2, you cannot do >>> np.int16('0xff', 16) either -- it's the same issue. It's also a minor issue, IMHO, as I doubt many people construct array scalars from strings, and even less do it in bases other than 10. The fix is to change array scalar __new__, but this is not completely straightforward to do. Patches are welcome. -- Pauli Virtanen From dwf at cs.toronto.edu Tue Aug 31 14:02:45 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Tue, 31 Aug 2010 14:02:45 -0400 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> On 2010-08-30, at 10:36 PM, Charles R Harris wrote: > I don't see what the connection with the determinant is. The log determinant will be calculated using the ordinary LU decomposition as that works for more general matrices. I think he means that if he needs both the determinant and to solve the system, it might be more efficient to do the SVD, obtain the determinant from the diagonal values, and obtain the solution by multiplying by U D^-1 V^T? David From friedrichromstedt at gmail.com Tue Aug 31 14:12:59 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Tue, 31 Aug 2010 20:12:59 +0200 Subject: [Numpy-discussion] Boolean arrays In-Reply-To: References: Message-ID: 2010/8/27 Brett Olsen : > If there's multiple possible valid values, I've come up with a couple > possible methods, but they all seem to be inefficient or kludges: >>>> valid = N.array(("a", "c")) >>>> (ar == valid[0]) | (ar == valid[1]) > array([ True, False, ?True, False, False, ?True, False, ?True, ?True], > dtype=bool) >>>> N.array(map(lambda x: x in valid, ar)) > array([ True, False, ?True, False, False, ?True, False, ?True, ?True], > dtype=bool) (ar[..., numpy.newaxis] == valid).T.sum(axis=0).T > 0 should also do the job. But it eats up memory. (It employs broadcasting.) Friedrich From aarchiba at physics.mcgill.ca Tue Aug 31 14:29:51 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Tue, 31 Aug 2010 14:29:51 -0400 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: Hi Melissa, On 30 August 2010 17:42, Melissa Mendon?a wrote: > I've been lurking for a while here but never really introduced myself. > I'm a mathematician in Brazil working with optimization and numerical > analysis and I'm looking into scipy/numpy basically because I want to > ditch matlab. Welcome to the list! I hope you will find the switch to numpy and scipy rewarding above and beyond not-being-matlab. Please feel free to ask questions on the list; as you've probably seen, we get lots of questions with simple but non-obvious answers, and a few really tough questions. > I'm just curious as to why you say "scipy.linalg.solve(), NOT > numpy.linalg.solve()". Can you explain the reason for this? I find > myself looking for information such as this on the internet but I > rarely find real documentation for these things, and I seem to have so > many performance issues with python... I'm curious to see what I'm > missing here. I agree that the documentation is a little hard to find one's way around sometimes. The numpy documentation project has done a wonderful job of providing detailed documentation for all the functions in numpy, but there's not nearly as much documentation giving a general orientation (which functions are better, what the relationship is with scipy and matplotlib). The scipy documentation project is unfortunately still getting started. What sorts of performance issues have you had? python has some important limitations, but there are often ways to work around them. Sometimes there isn't an efficient way to do things, though, and in those cases we'd like to think about whether numpy/scipy can be improved. (Bulk linear algebra - solving large numbers of small problems - stands out as an example.) Numerical optimization is another - we know the optimizers built into scipy have serious limitations, and welcome ideas on improving them. > Thanks, and sorry if I hijacked the thread, No problem, and welcome again to the list. Anne From eadrogue at gmx.net Tue Aug 31 15:13:50 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Tue, 31 Aug 2010 21:13:50 +0200 Subject: [Numpy-discussion] comparison between arrays of strings and numerical types In-Reply-To: References: <20100831163527.GA14551@doriath.local> Message-ID: <20100831191350.GA16581@doriath.local> 31/08/10 @ 09:44 (-0700), thus spake Keith Goodman: > 2010/8/31 Ernest Adrogu? : > > Hi, > > > > I find this a bit odd: > > > > In [18]: np.array(['a','b','c','d']) > 'a' > > Out[18]: array([False, ?True, ?True, ?True], dtype=bool) > > > > In [19]: np.array(['a','b','c','d']) > 4 > > Out[19]: True > > > > In [20]: np.array(['a','b','c','d']) > 4.5 > > Out[20]: True > > > > Is that right? I was expecting an element-wise comparison, > > but it returns a single truth value. > > Here's a recent discussion of the issue: > http://www.mail-archive.com/numpy-discussion at scipy.org/msg27051.html Ah, I suppose I wasn't paying attention :) There appears to be an open ticket as well: http://projects.scipy.org/numpy/ticket/725 Regards, Ernest From dwf at cs.toronto.edu Tue Aug 31 16:31:52 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Tue, 31 Aug 2010 16:31:52 -0400 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> Message-ID: <4FD7BF14-DD0D-442D-897D-150CE9B303C8@cs.toronto.edu> On 2010-08-30, at 10:19 PM, Dan Elliott wrote: > You don't think this will choke on a large (e.g. 10K x 10K) covariance matrix? That depends. Is it very close to being rank deficient?That would be my main concern. NumPy/LAPACK will have no trouble Cholesky-decomposing a matrix this big, presuming the matrix is well-behaved/full rank. Depending on your hardware it will be slow, but the Cholesky decomposition will be faster (I think usually by about a factor of 2) than other methods that do not assume positive-definiteness. At any rate, inverting the matrix explicitly will take longer and be quite a bit worse in terms of the eventual accuracy of your result. > Given what you know about how it computes the log determinant and how the > Cholesky decomposition, do you suppose I would be better off using eigen- > decomposition to do this since I will also get the determinant from the sum of > the logs of the eigenvalues? You could do it this way, though I am not certain of the stability concerns (and I'm in the middle of a move so my copy of Golub and Van Loan is packed up somewhere). I know that the SVD is used in numpy.leastsq, so it can't be that bad. I've never used covariance matrices quite so big that computing the determinant was a significant cost. One thing to note is that you should definitely not be solving the system for every single RHS vector separately. scipy.linalg.solve supports matrix RHSes, and will only do the matrix factorization (be it LU or Cholesky) once, requiring only the O(n^2) forward/backsubstitution to be done repeatedly. This will result in significant savings: In [5]: X = randn(10000,20000) In [6]: Y = dot(X, X.T) In [7]: timeit scipy.linalg.solve(Y, randn(10000), sym_pos=True) 10 loops, best of 3: 78.6 s per loop In [8]: timeit scipy.linalg.solve(Y, randn(10000, 50), sym_pos=True) 10 loops, best of 3: 81.6 s per loop David From jmccampbell at enthought.com Tue Aug 31 16:56:04 2010 From: jmccampbell at enthought.com (Jason McCampbell) Date: Tue, 31 Aug 2010 15:56:04 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Hi Chuck (and anyone else interested), I updated the refactoring page on the NumPy developer wiki (seems to be down or I'd paste in the link). It certainly isn't complete, but there are a lot more details about the data structures and memory handling and an outline of some additional topics that needs to be filled in. Regards, Jason On Wed, Aug 25, 2010 at 10:20 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Wed, Aug 25, 2010 at 8:19 AM, Jason McCampbell < > jmccampbell at enthought.com> wrote: > >> Chuck, >> >> I will update the wiki page on the Numpy developer site that discusses the >> refactoring this week. Right now what's there reflects our plans before >> they met the reality of code. Needless to say, the actual implementation >> differs in some of the details. >> >> Here is a very brief overview of the structure: >> >> - The libndarray directory now contains all of the code for the 'core' >> library. This library is independent of Python and implements most of the >> array, descriptor, iterator, and ufunc functionality. The goal is that all >> non-trivial behavior should be in here, but in reality some parts are tied >> fairly tightly to the CPython interpreter and will take more work to move >> into the core. >> >> - numpy/core/src/multiarray and numpy/core/src/umath now implement "just" >> the CPython interface to libndarray. We have preserved both the Python >> interface and the C API. Ideally each C API function is just a simple >> wrapper around a call to the core API, though it doesn't always work out >> that way. However, a large amount of code has been moved out of these >> modules into the core. >> >> - The core is built as a shared library that is independent of any given >> interface layer. That is, the same shared library/DLL can be used with >> CPython, IronPython and any other implementation. Each interface is >> required to pass in a set of callbacks for handling reference counting, >> object manipulation, and other interface-specific behavior. >> >> - The core implements its own reference counting semantics that happen to >> look very much like CPython's. This was necessary to make the core library >> independent of the interface layer and preserve performance (ie, limit the >> number of callbacks). The handshaking between interface and core is a bit >> complicated but ends up working nicely and efficiently for both reference >> counted and garbage collected systems. I'll write up the details on the >> wiki page. >> >> - As Travis mentioned we are also working on a .NET back end to Cython. >> This lets us port the modules such as MTRAND without having to have two >> separate interfaces, a Cython and a .NET version. Instead, we can modify >> the existing .pyx file to use the new core API (should improve performance >> in CPython version slightly). Once done, Cython can generate the .NET and >> CPython interfaces from the same .pyx file. >> >> We have done a fair amount of cleanup on the naming conventions but >> certainly more needs to be done! >> >> I'll write it up for everyone this week but feel free to email me with >> other questions. >> >> > Thanks for the summary, it clarifies things a lot. On my cleanup wish list, > some of the functions use macros that contain jumps, which is not so nice. > I've been intending to scratch that itch for several years now but haven't > gotten around to it. I expect such things have a lower priority than getting > the basic separation of functionality in place, but just in case... > > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danelliottster at gmail.com Tue Aug 31 18:52:58 2010 From: danelliottster at gmail.com (Dan Elliott) Date: Tue, 31 Aug 2010 22:52:58 +0000 (UTC) Subject: [Numpy-discussion] inversion of large matrices References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> Message-ID: David Warde-Farley cs.toronto.edu> writes: > On 2010-08-30, at 10:36 PM, Charles R Harris wrote: > I think he means that if he needs both the determinant and to solve the > system, it might be more efficient to do > the SVD, obtain the determinant from the diagonal values, and obtain the > solution by multiplying by U D^-1 V^T? Thank you, that is what I meant. Poorly worded on my part. In particular, I am writing code to invert a very large covariance matrix. I think David has some good information in another post in this thread. - dan From charlesr.harris at gmail.com Tue Aug 31 19:58:30 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 Aug 2010 17:58:30 -0600 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> Message-ID: On Tue, Aug 31, 2010 at 4:52 PM, Dan Elliott wrote: > David Warde-Farley cs.toronto.edu> writes: > > On 2010-08-30, at 10:36 PM, Charles R Harris wrote: > > I think he means that if he needs both the determinant and to solve the > > system, it might be more efficient to do > > the SVD, obtain the determinant from the diagonal values, and obtain the > > solution by multiplying by U D^-1 V^T? > > Thank you, that is what I meant. Poorly worded on my part. > > In particular, I am writing code to invert a very large covariance matrix. > I > think David has some good information in another post in this thread. > > Where did the covariance array come from? It may be the case that you can use a much smaller one, for instance in PCA of images. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Tue Aug 31 23:31:41 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 31 Aug 2010 20:31:41 -0700 Subject: [Numpy-discussion] Where is the dev version of numpydoc? Message-ID: Hello, I would like to update my numpydoc so it works with sphinx 1.0, but I am not sure where the dev version is; can someone point me in the right direction? John -------------- next part -------------- An HTML attachment was scrubbed... URL: