[Tutor] same python script now running much slower

[Protas, Meredith]

Thanks for all of your comments!  I am working with human genome information which is in the form of many very short DNA sequence reads.  I am using a script that sorts through all of these sequences and picks out ones that contain a particular sequence I'm interested in.  Because my data set is so big, I have the data on an external hard drive (but that's where I had it before when it was faster too).

As for how much slower it is running, I don't know because I keep having to move my computer before it is finished.  The size of the data is the same, the script has not been modified, and the data is still in the same place.  Essentially, I'm doing exactly what I did before (as a test) but it is now slower.

How would I test your suggestion, Bill, that the script is paging itself to death?  The data has not grown and I don't think the number of processes occupying memory has changed.

By the way, I am using a Mac and I've tried two different computers.

Thanks so much for all of your help!

Meredith

Sent from my iPhone

On Dec 30, 2013, at 2:37 PM, "William Ray Wing" <wrw at mac.com<mailto:wrw at mac.com>> wrote:

On Dec 30, 2013, at 1:37 PM, "Protas, Meredith" <ProtasM at vision.ucsf.edu<mailto:ProtasM at vision.ucsf.edu>> wrote:

Hi,

I'm very new to python so I'm sorry about such a basic question.

I am using a python script generated by another person.  I have used this script multiple times before and it takes around 24 hours to run.  Recently, I have tried to run the script again (the same exact command lines) and it is much much slower.  I have tried on two different computers with the same result.  I used top to see if there were any suspicious functions that were happening but there seems to not be.  I also ran another python script I used before and that went at the same speed as before so the problem seems unique to the first python script.

Does anyone have any idea why it is so much slower now than it used to be (just around a month ago).

Thanks for your help!

Meredith

Meredith,  This is just a slight expansion on the note you received from Alan.  Is there any chance that the script now is paging itself to death?  That is, if you are reading a huge amount of data into a structure in memory, and if it no longer fits in available physical memory (either because the amount of data to be read has grown or the number of other processes that are occupying memory have grown), then that data structure may have gone virtual and the OS may be swapping it out to disk.  That would dramatically increase the amount of elapsed wall time the program takes to run.

If you can tell us more about what the program actually is doing or calculating, we might be able to offer more help.

-Bill
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20131231/c168f148/attachment.html>