How to use a 5 or 6 bit integer in Python?

[Rainer Deyke]

Glen Wheeler wrote:
>   Hello all,
>
>   My program uses many millions of integers, and Python is allocating
> way too much memory for these.  I can't have the performance hit by
> using disk, so I figured I'd write a C extension to define a new type.
>   Problem is, my C knowledge is years old and regardless of my
> attempts distutils will not recognise my installation of the MS
> compiler.
>   I am thinking, is there any easier way to use a 5 or 6 bit integer
> in python?  Even a regular 8-bit would be fine.  I only need to
> represent either 32 or 64 distinct numbers.

You're trying to solve the wrong problem.  Python caches small integers, so
you've only got a very small number of distinct integer objects.  This is a
good thing, because each Python object has a eight byte overhead.
Introducing a new type wouldn't result in any memory savings.

The real overhead comes from the references to those objects.  Each
reference is four bytes (on 32 bit computers), and there is no way to
decrease this.  Your best option is to use high level objects that contain
many integers internally but don't store them as individual Python
references.  The array module is one choice, Numarray is another.  If
immutability isn't an issue, you could even use strings (with each character
treated as an eight bit integer).


-- 
Rainer Deyke - rainerd at eldwood.com - http://eldwood.com