[Numpy-discussion] Numpy-discussion Digest, Vol 32, Issue 39
Emerald Jasper
emerald.jasper at yahoo.co.uk
Thu May 14 07:06:37 EDT 2009
Hi,
Actually, I am quite new in programming, so could you please send me the syntax so that I can use in my research.
thank you so much.
--- On Thu, 14/5/09, numpy-discussion-request at scipy.org <numpy-discussion-request at scipy.org> wrote:
From: numpy-discussion-request at scipy.org <numpy-discussion-request at scipy.org>
Subject: Numpy-discussion Digest, Vol 32, Issue 39
To: numpy-discussion at scipy.org
Date: Thursday, 14 May, 2009, 10:51 AM
Send Numpy-discussion mailing list submissions to
numpy-discussion at scipy.org
To subscribe or unsubscribe via the World Wide Web, visit
http://mail.scipy.org/mailman/listinfo/numpy-discussion
or, via email, send a message with subject or body 'help' to
numpy-discussion-request at scipy.org
You can reach the person managing the list at
numpy-discussion-owner at scipy.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Numpy-discussion digest..."
Today's Topics:
1. Re: numpy slices limited to 32 bit values? (Glenn Tarbox, PhD)
2. Re: numpy slices limited to 32 bit values? (Gael Varoquaux)
3. Re: numpy slices limited to 32 bit values? (Glenn Tarbox, PhD)
4. Re: numpy slices limited to 32 bit values? (Gael Varoquaux)
5. Re: (no subject) (Sebastian Walter)
6. Non-linear optimization in python (Emerald Jasper)
7. Re: Non-linear optimization in python (Matthieu Brucher)
----------------------------------------------------------------------
Message: 1
Date: Thu, 14 May 2009 01:31:45 -0700
From: "Glenn Tarbox, PhD" <glenn at tarbox.org>
Subject: Re: [Numpy-discussion] numpy slices limited to 32 bit values?
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID:
<f049b35c0905140131t12efe2ddgde3285d7ade721c1 at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
On Wed, May 13, 2009 at 11:22 PM, Glenn Tarbox, PhD <glenn at tarbox.org>wrote:
>
>
> On Wed, May 13, 2009 at 11:04 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, May 13, 2009 at 10:50 PM, Glenn Tarbox, PhD <glenn at tarbox.org>wrote:
>>
>>> I'm using the latest version of Sage (3.4.2) which is python 2.5 and
>>> numpy something or other (I will do more digging presently)
>>>
>>> I'm able to map large files and access all the elements unless I'm using
>>> slices
>>>
>>> so, for example:
>>>
>>> fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64',
>>> mode='r+', shape=(10000000000,))
>>>
>>> which is 1e10 doubles if you don't wanna count the zeros
>>>
>>> gives full access to a 75 GB memory image
>>>
>>> But when I do:
>>>
>>> fp[:] = 1.0
>>> np.sum(fp)
>>>
>>> I get 1410065408.0 as the result
>>>
>>
>> As doubles, that is more than 2**33 bytes, so I expect there is something
>> else going on. How much physical memory/swap memory do you have? This could
>> also be a python problem since python does the memmap.
>>
>
> I've been working on some other things lately and that number seemed
> related to 2^32... now that I look more closely, I don't know where that
> number comes from.
>
> To your question, I have 32GB of RAM and virtually nothing else running...
> Top tells me I'm getting between 96% and 98% for this process which seems
> about right.
>
> Here's the thing. When I create the mmap file, I get the right number of
> bytes. I can, from what I can tell, update individual values within the
> array (I'm gonna bang on it a bit more with some other scripts)
>
> Its only when using slicing that things get strange (he says having not
> really done a more thorough test)
>
> Of course, I was assuming this is a 32 bit thing... but you're right...
> where did that result come from???
>
> The other clue here is that when I create my own slice (as described above)
> it returns instantly... numpy doesn't throw an error but it doesn't do
> anything with the slice either.
>
> Since I'm IO bound anyways, maybe i'll just write a loop and see if I can't
> set all the values. The machine could use a little exercise anyways.
>
I ran the following test:
import numpy as np
size=10000000000
fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64',
mode='r+', shape=(size,))
for i in xrange(size):
fp[i]=1.0
time np.sum(fp)
10000000000.0
Time: CPU 188.36 s, Wall: 884.33 s
So, everything seems to be working and it kinda makes sense. The sum
should be IO bound which it is. I didn't time the loop but it took a while
(maybe 30 minutes) and it was compute bound.
To make sure, I exited the program and ran everything but the initialization
loop.
import numpy as np
size=10000000000
fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64',
mode='r+', shape=(size,)
time np.sum(fp)
10000000000.0
Time: CPU 180.02 s, Wall: 854.72 s
I was a little surprised that it didn't take longer given almost half of the
mmap'ed data should have been resident in the sum performed immediately
after initialization, but since it needed to start at the beginning and only
had the second half in memory, it makes sense
So, it "appears" as though the mmap works but there's something strange with
slices going on.
-glenn
>
>
>>
>> Chuck
>>
>>
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
> Glenn H. Tarbox, PhD || 206-274-6919
> http://www.tarbox.org
>
--
Glenn H. Tarbox, PhD || 206-274-6919
http://www.tarbox.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090514/c58e77fd/attachment-0001.html
------------------------------
Message: 2
Date: Thu, 14 May 2009 10:43:35 +0200
From: Gael Varoquaux <gael.varoquaux at normalesup.org>
Subject: Re: [Numpy-discussion] numpy slices limited to 32 bit values?
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID: <20090514084335.GD32437 at phare.normalesup.org>
Content-Type: text/plain; charset=utf-8
On Thu, May 14, 2009 at 01:31:45AM -0700, Glenn Tarbox, PhD wrote:
> I've been working on some other things lately and that number seemed
> related to 2^32... now that I look more closely, I don't know where that
> number comes from.
Is your OS 64bit?
Ga?
------------------------------
Message: 3
Date: Thu, 14 May 2009 02:13:23 -0700
From: "Glenn Tarbox, PhD" <glenn at tarbox.org>
Subject: Re: [Numpy-discussion] numpy slices limited to 32 bit values?
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID:
<f049b35c0905140213m7c82a374i566e961f3ed78c60 at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
On Thu, May 14, 2009 at 1:43 AM, Gael Varoquaux <
gael.varoquaux at normalesup.org> wrote:
> On Thu, May 14, 2009 at 01:31:45AM -0700, Glenn Tarbox, PhD wrote:
> > I've been working on some other things lately and that number seemed
> > related to 2^32... now that I look more closely, I don't know where
> that
> > number comes from.
>
> Is your OS 64bit?
Yes, Ubuntu 9.04 x86_64
Linux hq2 2.6.28-11-server #42-Ubuntu SMP Fri Apr 17 02:45:36 UTC 2009
x86_64 GNU/Linux
-glenn
>
> Ga?l
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
--
Glenn H. Tarbox, PhD || 206-274-6919
http://www.tarbox.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090514/5ae4b5a2/attachment-0001.html
------------------------------
Message: 4
Date: Thu, 14 May 2009 11:16:17 +0200
From: Gael Varoquaux <gael.varoquaux at normalesup.org>
Subject: Re: [Numpy-discussion] numpy slices limited to 32 bit values?
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID: <20090514091617.GF32437 at phare.normalesup.org>
Content-Type: text/plain; charset=utf-8
On Thu, May 14, 2009 at 02:13:23AM -0700, Glenn Tarbox, PhD wrote:
> On Thu, May 14, 2009 at 1:43 AM, Gael Varoquaux
> <[1]gael.varoquaux at normalesup.org> wrote:
> On Thu, May 14, 2009 at 01:31:45AM -0700, Glenn Tarbox, PhD wrote:
> > ? ? ?I've been working on some other things lately and that number
> seemed
> > ? ? ?related to 2^32... now that I look more closely, I don't know
> where that
> > ? ? ?number comes from.
> Is your OS 64bit?
> Yes, Ubuntu 9.04 x86_64
Hum, I am wondering: could it be that Sage has not been compiled in
64bits? That number '32' seems to me to point toward a 32bit pointer
issue (I may be wrong).
Ga?
------------------------------
Message: 5
Date: Thu, 14 May 2009 11:19:03 +0200
From: Sebastian Walter <sebastian.walter at gmail.com>
Subject: Re: [Numpy-discussion] (no subject)
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID:
<ec9f80fa0905140219y185b7d99neb2e318abd2c216e at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On Wed, May 13, 2009 at 10:18 PM, David J Strozzi <strozzi2 at llnl.gov> wrote:
> Hi,
>
> [You may want to edit the numpy homepage numpy.scipy.org to tell
> people they must subscribe to post, and adding a link to
> http://www.scipy.org/Mailing_Lists]
>
>
> Many of you probably know of the interpreter yorick by Dave Munro. As
> a Livermoron, I use it all the time.
Never heard of it... what does it do? By the sound of it, yorick is an
interpreted language like Python.
> There are some built-in
> functions there, analogous to but above and beyond numpy's sum() and
> diff(), which are quite useful for common operations on gridded data.
> Of course one can write their own, but maybe they should be cleanly
> canonized?
>
> For instance:
>
> x = linspace(0,10,10)
> y = sin(x)
>
> It is common, say when integrating y(x), to take "point-centered"
> data and want to zone-center it:
>
> I = sum(zcen(y)*diff(x))
>
> def zcen(x): return 0.5*(x[0:-1]+x[1:])
>
> Besides zcen, yorick has builtins for "point centering", "un-zone
> centering," etc. Also, due to its slick syntax you can give these
> things as array "indexes":
>
> x(zcen), y(dif), z(:,sum,:)
>
>
> Just some thoughts,
> David Strozzi
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
------------------------------
Message: 6
Date: Thu, 14 May 2009 10:26:25 +0000 (GMT)
From: Emerald Jasper <emerald.jasper at yahoo.co.uk>
Subject: [Numpy-discussion] Non-linear optimization in python
To: numpy-discussion at scipy.org
Message-ID: <342303.58705.qm at web23902.mail.ird.yahoo.com>
Content-Type: text/plain; charset="utf-8"
Dear python user!
Please, instruct me how to make non-linear optimization using numpy/simpy in python?
Thank you very much in the advance,
Emerald
from Japan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090514/5ef19c2b/attachment-0001.html
------------------------------
Message: 7
Date: Thu, 14 May 2009 12:51:29 +0200
From: Matthieu Brucher <matthieu.brucher at gmail.com>
Subject: Re: [Numpy-discussion] Non-linear optimization in python
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Message-ID:
<e76aa17f0905140351l7e276c9bw8f7bb488d7fa03c9 at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi,
You have several choices:
- using scipy.optimize
- openopt
- the old openopt scikit that contains a generic optimization framework.
Did you try one of these?
2009/5/14 Emerald Jasper <emerald.jasper at yahoo.co.uk>:
> Dear python user!
> Please, instruct me how to make non-linear optimization using numpy/simpy in
> python?
> Thank you very much in the advance,
> Emerald
> from Japan
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
--
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher
------------------------------
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
End of Numpy-discussion Digest, Vol 32, Issue 39
************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090514/dccd4a55/attachment.html>
More information about the NumPy-Discussion
mailing list