[Tutor] is this use or abuse of __getitem__ ?

[eryksun]

On Sat, Sep 15, 2012 at 10:18 AM, Albert-Jan Roskam <fomcl at yahoo.com> wrote:

> Thanks, I hadn't noticed this yet. I am refactoring some of the rest of my code
> and I hadn't run anything yet. My code has two methods that return record(s):
> an iterator (__getitem__) and a generator (readFile, which is also called by
> __enter__). Shouldn't I also take the possibility of a MemoryError into
> account when the caller does something like data[:10**8]? It may no longer fit
> into memory, esp. when the dataset is also wide.

The issue with c_long isn't a problem for a slice since
key.indices(self.nCases) limits the upper bound. For the individual
index you had it right the first time by raising IndexError before it
even gets to the c_long conversion. I'm sorry for wasting your time on
a non-problem. However, your test there is a bit off. A negative index
can be -nCases since counting from the end starts at -1. If you first
do the ternary check to add the offset to a negative index, afterward
you can raise an IndexError if "not 0 <= value < nCases".

As to MemoryError, dealing with gigabytes of data in main memory is
not a problem I've come up against in practice. You might still want a
reasonable upper bound for slices. Often when the process runs out of
memory it won't even see a MemoryError. The OS simply kills it. On the
other hand, while bugs like a c_long wrapping around need to be caught
to prevent silent corruption of data, there's nothing at all silent
about crashing the process. It's up to you how much you want to
micromanage the situation. You might want to check out psutil as a
cross-platform way to monitor the process memory usage:

http://code.google.com/p/psutil

If you're also supporting the iterator protocol with the __iter__
method, then I think a helper _items(start, stop, step) generator
function would be a good idea.

Here's an updated example (not tested however; it's just a suggestion):


    import operator

    def _items(self, start=0, stop=None, step=1):
        if stop is None:
            stop = self.nCases

        for i in range(start, stop, step):
            retcode1 = self.iomodule.SeekNextCase(self.fh, ctypes.c_long(i))
            self.caseBuffer, self.caseBufferPtr = self.getCaseBuffer()
            retcode2 = self.iomodule.WholeCaseIn(self.fh, self.caseBufferPtr)
            record = struct.unpack(self.structFmt, self.caseBuffer.raw)
            if any([retcode1, retcode2]):
                raise RuntimeError("Error retrieving record %d [%s, %s]" %
                    (i, retcodes[retcode1], retcodes[retcode2]))
            yield record


    def __iter__(self):
        return self._items()


    def __getitem__(self, key):

        is_slice = isinstance(key, slice)

        if is_slice:
            start, stop, step = key.indices(self.nCases)
        else:
            key = operator.index(key)
            start = key + self.nCases if key < 0 else key
            if not 0 <= start < self.nCases:
                raise IndexError
            stop = start + 1
            step = 1

        records = self._items(start, stop, step)
        if is_slice:
            return list(records)
        return next(records)


> but I didn't know about LOAD_ATTR.

That's the bytecode operation to fetch an attribute. Whether or not
bypassing it will provide a significant speedup depends on what else
you're doing in the loop. If the the single LOAD_ATTR is only a small
fraction of the total processing time, or you're not looping thousands
of times, then this little change is insignificant.


> Is a list comprehension still faster than this?

I think list comprehensions or generator expressions are best if the
evaluated expression isn't too complex and uses built-in types and
functions. I won't typically write a function just to use a list
comprehension for a single statement. Compared to a regular for loop
(especially if append is cached in a fast local), the function call
overhead makes it a wash or worse, even given the comprehension's
efficiency at building the list. If the main work of the loop is the
most significant factor, then the choice of for loop vs list
comprehension doesn't matter much with regard to performance, but I
still think it's simpler to just use a regular for loop. You can also
write a generator function if you need to reuse an iteration in
multiple statements.

> Does it also mean that e.g. "from ctypes import *" (--> c_long()) is
> faster than "import ctypes" (--> ctypes.c_long()). I am now putting as much as
> possible in __init__. I don't like the first way of importing at all.

It's not a good idea to pollute your namespace with "import *"
statements. In a function, you can cache an attribute locally if doing
so will provide a significant speedup. Or you can use a default
argument like this:

    def f(x, c_long=ctypes.c_long):
        return c_long(x)