[Python-bugs-list] [ python-Bugs-472568 ] PyBuffer_New() memory not aligned

Mon Sep 22 14:57:11 EDT 2003

Bugs item #472568, was opened at 2001-10-18 21:31
Message generated for change (Comment added) made by arigo
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=472568&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: PyBuffer_New() memory not aligned

Initial Comment:
Memory buffer areas created by PyBuffer_New are

missaligned on 32 bit machines, for doubles (64 bit).

This typically generates a bus error crash on most RISC

machines when a library function tries to write a

double in a memory buffer allocated by PyBuffer_New().

When looking at the bufferobject.c code, this seems to

come from the fact that  the memory buffer points at

'malloc() + sizeof  struct PyBufferObject'; 	[line:

b->b_ptr = (void *)(b + 1); ]

To the extent that struct PyBufferObject is not a

multiple of sizeof (double), the largest value type;

thus the misalignment, and the crashes on most RISC

processors.

A quick and temporary solution would consist in adding

a dummy double field as the last field of the

PyBufferObject structure:

struct PyBufferObject {

...

double dummy; 

} ;

and setting

b->b_ptr = (void*)&b->dummy;  /*was (void*(b+1)*/

In doing so, b->b_ptr will always by aligned on sizeof

(double), on either 32 and 64 bit architectures.

Since I'm on the buffer type problem: It would be nice

(and probably easy to do) to augment the buffer

protocol interface with the ability to specify the

basic value type stored in the buffer.

A possible list of values (enum ...) would be:

- undefined (backward compatibility)

- char, unsigned char, short, ....int, ... long, long

long, float, double, and void* (memory address).

This would enable to check at runtime the type of

values stored in a buffer (and prevent missalignement

buserrors, as well as catching garbage situations when

improper array types are passed by means of the buffer

interface [e.g.: int/float/double/short...).

Frederic Giacometti

Frederic Giacometti

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2003-09-22 18:57

Message:
Logged In: YES 
user_id=4771

No more activity around here. I suggest to deprecate this

bug report. Here is a note for the docs saying the memory

PyBuffer_New() gets you isn't specifically aligned. (cannot

attach it, sorry)

http://arigo.tunes.org/api_concrete.diff

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-06-17 15:30

Message:
Logged In: YES 
user_id=31435

Unassigned -- I don't expect to work on this.

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-11-01 17:21

Message:
Logged In: YES 
user_id=93657

The solution Tim proposes (2d malloc) will be very fine with

what we do.

Here are some more details on what we're doing (and this is

a standard operation):

- We want to create an array of double that we pass to a C

function, and then return this array to Python as buffer

object (the buffer is passed latter on as arg to other

functions using the buffer interface); and do it so that

Python takes ownership of the buffer memory management.

- We don't want to require Numerical to operate the package;

just for memory allocation.

- We should actually be using the Python array module

interface.

Unfortunately:

  * the Python array object C definitions are not exported

in a .h file

  * the Array python interface does not provide the ability

to create a new array of an arbitrary size (and certainly

initialized to 0). One has to provide a list or a string to

create an array of a given size.  IThis is not workable if

we work we large arrays (e.g.: an array of 1.000.000 doubles

is only 8 MB RAM ...).

Another  solution, then, would consist in extending the

array Python interface, so as to enable the creation of

arrays of arbitrary sizes (prefereably initialized to 0 or

to something alse with a calloc or a memset).

The extension of the array.array() function could be the

better solution, taking into account our needs as well as

Tim's concerns.

FG

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-01 01:31

Message:
Logged In: YES 
user_id=31435

The only portable way to fix this (assuming it's broken -- 

I don't see any alignment promises in the docs, and since 

we never use it we can't mine the source code for clues 

either) is to have PyBuffer_New do a second separate malloc

(size) and set b_ptr to that.  The C std guarantees memory 

returned by malloc is suitably aligned for all purposes; it 

doesn't promise that any standard type captures the 

strictest alignment requirement (indeed, at KSR we returned 

128-byte aligned memory from malloc, to cater to 

our "subpage" type extension).

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-11-01 00:31

Message:
Logged In: YES 
user_id=93657

A portable solution (im[provement over what I proposed)

would consitst in declaring 'dummy' with a union type,

'unionizing' all C-ANSI value types (and including 'long

long' optionally by mean of an #ifdef).

 { .... union { int Int; long Long; double Double; void*

Pvoid ...} dummy; }

All (void*)obj->dummy can be replaced with obj->dummy.Pvoid

FG

----------------------------------------------------------------------

Comment By: Jeremy Hylton (jhylton)
Date: 2001-10-22 20:19

Message:
Logged In: YES 
user_id=31392

Note to whomever take this bug: PyBuffer_New() is not called

anywhere in the Python source tree; nor are there any tests

for buffer objects that I'm aware of.  A few simple test

cases would have caught this bug already.  (And for the case

of the builtin buffer() call, it might be good if it used

PyBuffer_New().)

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-10-18 21:35

Message:
Logged In: YES 
user_id=93657

I wasn't looged in when I submitted the item. Don't think

I'm becoming anonymous :))

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=472568&group_id=5470