Best way to inline a primitive array in RPython?

Looking at rbuffer, rgc, and the lltype namespaces, it's not completely clear to me how I would go about writing a struct with a resizable primitive array inlined. Something like: typedef struct foo_t { int a, b; int _data[0]; } foo_t tmp = malloc(sizeof(foo_t) + 64); I can see that the GC has the ability to allocate "extra" memory beyond what's required for a struct, and that lltype has the ability to inline some lists into a struct, but how does this work in practice? And maybe I'm missing something in the docs, so simply pointing me to some docs is also fine. -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth)

Hi Timothy/ The code below is kind of a C-trick (or hack) based on the knowledge of how the C-compiler lays out structs in memory. In your example it would be: struct layout +------+ | a +------+ | b +------+ | _data[0] +------+ Adding the +64, gives you 64 extra bytes (64/4) = 16 int's if sizeof(int) is 4 (32-bit)), making the layout look like: +------+ | a +------+ | b +------+ | _data[0] +------+ | +1 +------+ | +2 +------+ | . . . +------+ | +16 +------+ So now you can safely address tmp._data[15] = 0x12345678 ; // last element in struct I don't know your need, but I don't think C was intended to be used like this, best practice is: 1) Hard allocate memory #define SIZE 16 typedef struct foo_t { int a, b; int _data[SIZE]; } 2) Dynamic allocate memory typedef struct foo_t { int a, b; int* _data; struct tmp = foo_t; tmp._data = malloc(64) 3) Hack alloc - see above :-)
malloc() returns a pointer, so It should be foo_t* tmp = (foo_t*) malloc(sizeof(foo_t) + 64);
I can see that the GC has the ability to allocate "extra" memory beyond what's required for a struct, and that lltype has the ability to inline some lists into a struct, but how does this work in practice? And maybe I'm missing something in the docs, so simply pointing me to some docs is also fine.
Not sure where to find this, it is sometimes used, but I am not sure I would recommend it, unless you know exactly what your are doing and like keeping track of allocated memory :-) br /Rene

Hi Timothy, On Tue, 30 Oct 2018 at 00:42, Timothy Baldridge <tbaldridge@gmail.com> wrote:
You can do that if you use the lltype.GcStruct type directly, not using "regular" RPython code. See the main example in rpython.rtyper.lltypesystem.rstr: the types STR and UNICODE. They are defined as lltype.GcStruct('name', ..some_regular_fields.., inlined_array_field), and allocated with lltype.malloc(STR, length_of_array). Note that you also need to prevent the JIT from seeing any such type: it has got special case for STR and UNICODE but doesn't handle the general case. A bientôt, Armin.

Hi Timothy/ The code below is kind of a C-trick (or hack) based on the knowledge of how the C-compiler lays out structs in memory. In your example it would be: struct layout +------+ | a +------+ | b +------+ | _data[0] +------+ Adding the +64, gives you 64 extra bytes (64/4) = 16 int's if sizeof(int) is 4 (32-bit)), making the layout look like: +------+ | a +------+ | b +------+ | _data[0] +------+ | +1 +------+ | +2 +------+ | . . . +------+ | +16 +------+ So now you can safely address tmp._data[15] = 0x12345678 ; // last element in struct I don't know your need, but I don't think C was intended to be used like this, best practice is: 1) Hard allocate memory #define SIZE 16 typedef struct foo_t { int a, b; int _data[SIZE]; } 2) Dynamic allocate memory typedef struct foo_t { int a, b; int* _data; struct tmp = foo_t; tmp._data = malloc(64) 3) Hack alloc - see above :-)
malloc() returns a pointer, so It should be foo_t* tmp = (foo_t*) malloc(sizeof(foo_t) + 64);
I can see that the GC has the ability to allocate "extra" memory beyond what's required for a struct, and that lltype has the ability to inline some lists into a struct, but how does this work in practice? And maybe I'm missing something in the docs, so simply pointing me to some docs is also fine.
Not sure where to find this, it is sometimes used, but I am not sure I would recommend it, unless you know exactly what your are doing and like keeping track of allocated memory :-) br /Rene

Hi Timothy, On Tue, 30 Oct 2018 at 00:42, Timothy Baldridge <tbaldridge@gmail.com> wrote:
You can do that if you use the lltype.GcStruct type directly, not using "regular" RPython code. See the main example in rpython.rtyper.lltypesystem.rstr: the types STR and UNICODE. They are defined as lltype.GcStruct('name', ..some_regular_fields.., inlined_array_field), and allocated with lltype.malloc(STR, length_of_array). Note that you also need to prevent the JIT from seeing any such type: it has got special case for STR and UNICODE but doesn't handle the general case. A bientôt, Armin.
participants (3)
-
Armin Rigo
-
Rene Nejsum
-
Timothy Baldridge