[ python-Feature Requests-1023290 ] proposed struct module format code addition

[SourceForge.net]

Feature Requests item #1023290, was opened at 2004-09-06 13:42
Message generated for change (Settings changed) made by josiahcarlson
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Josiah Carlson (josiahcarlson)
>Assigned to: Raymond Hettinger (rhettinger)
Summary: proposed struct module format code addition

Initial Comment:
I believe there should be a mechanism to load and
unload arbitrarily large integers via the struct
module.  Currently, one would likely start with the 'Q'
format character, creating the integer in a block-wise
fashion with multiplies and shifts.

This is OK, though it tends to lend itself to certain
kinds of bugs.

There is currently another method for getting large
integers from strings and going back without the struct
module:

long(stri.encode('hex'), 16)
hex(inte)[2:].decode('hex')


Arguably, such things shouldn't be done for the packing
and unpacking of binary data in general (the string
slicing especially).


I propose a new format character for the struct module,
specifically because the struct module is to "Interpret
strings as packed binary data".  Perhaps 'g' and 'G'
(eg. biGint) is sufficient, though any reasonable
character should suffice.  Endianness should be
handled, and the number of bytes representing the
object would be the same as with the 's' formatting
code.  That is, '>60G' would be an unsigned big-endian
integer represented by 60 bytes (null filled if the
magnitude of the passed integer is not large enough).

The only reason why one wouldn't want this
functionality in the struct module is "This module
performs conversions between Python values and C
structs represented as Python strings." and arbitrarily
large integers are not traditionally part of a C struct
(though I am sure many of us have implemented arbitrary
precision integers with structs).  The reason "not a C
type" has been used to quash the 'bit' and 'nibble'
format character, because "masks and shifts" are able
to emulate them, and though "masks and shifts" could
also be used here, I have heard myself and others state
that there should be an easy method for converting
between large longs and strings.


A side-effect for allowing arbitrarily large integers
to be represented in this fashion is that its
functionality could, if desired, subsume the other
integer type characters, as well as fill in the gaps
for nonstandard size integers (3, 5, 6, 7 etc. byte
integers), that I (and I am sure others) have used in
various applications.


Currently no implementation exists, and I don't have
time to do one now.  Having taken a look at
longobject.c and structmodule.c, I would likely be able
to make a patch to the documentation, structmodule.c,
and test_struct.py around mid October, if this
functionality is desireable to others and accepted. 
While I doubt that a PEP for this is required, if
necessary I would write one up with a sample
implementation around mid October.

----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2004-10-05 18:59

Message:
Logged In: YES 
user_id=139309

I would definitely have an immediate use for 3 byte integers.. the Mach-
O executable format has a couple fields that are 3 byte unsigned 
integers (bit flags).  py2app's supporting library macholib reads and 
writes this format directly.  Currently I have several places that look 
like this:

class dylib_reference(Structure):
    _fields_ = (
        # XXX - ick, fix
        ('isym_flags', p_ulong),
        #('isym', p_ubyte * 3),
        #('flags', p_ubyte),
    )

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-10-02 16:00

Message:
Logged In: YES 
user_id=80475

If no one other that the OP supports this, I would like to
reject putting this in the struct module.  Initially, it
seemed like a fit because the endian options and whatnot are
already in place; however, in one way or another each of the
posters except the OP has stated good reasons for it not
being in the struct module.

Variable length C structure members are not what the module
is about. Having to know the length in advance of the call
is a killer.  The learning curve issues with struct are also
a problem.  And, the use cases jsut don't point to struct.

Put in a simple function in binascii or let's drop it.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-10-02 15:34

Message:
Logged In: YES 
user_id=341410

I have just attached a unified diff against structmodule.c
2.62 in CVS.

It implements the semantics I have been describing, compiles
cleanly, and produces proper results.

>>> pickle.encode_long(83726)
'\x0eG\x01'
>>> struct.pack('<3g', 83726)
'\x0eG\x01'
>>> struct.unpack('<3g', struct.pack('<3g', 83726))
(83726L,)

If the functionality is accepted, I will submit diffs for
test_struct.py and libstruct.tex .

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-16 11:11

Message:
Logged In: YES 
user_id=341410

Curse you Tim, for your core Python experience *wink*.

Pickle is one example where a pascal-like encoding of longs
was an encoding decision made to be flexible and space
efficient.

Certainly we have disparate use cases.  Mine is for fixed
struct-like records with multiple types.  With pickle, any
/thought/ of fixed records are tossed out the window with
variable-lengthed types like strings, longs, lists, tuples
and dicts, and I believe aren't really comparable.  Now,
variable-lengthed longs packed in little-endian format
already have a mechanism for encoding and decoding via
pickle.en/decode_long (though it is wholly undocumented),
and seemingly is going to get another in binascii. 
Fixed-lengthed, optional signed/unsigned, optional
little-endian/big-endian longs do not have a mechanism for
encoding and decoding, which is what I am asking for.

I will point out that 128 bit integers are gaining support
on newer 32 and 64 bit processors and C compilers for them
(SSE on x86, Itanium, etc.).  In the future, a new code for
these 128 bit integers may be asked for inclusion.  With a
variable-width integer type, all future "hey, we now have
x-byte types in C, where is struct support in Python?", can
be answered with the proposed, "choose your integer size"
format code.  That is to say, this format code is future
proof, unless integer types start wandering from integer
byte widths.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-09-15 19:52

Message:
Logged In: YES 
user_id=31435

Use cases are important.  Oddly(?) enough, I've never had a 
need for a bigint conversion in a C struct, and have a hard 
time imagining I will someday.  All the cases I've had (and I've 
had more than a few) were one-shot str->long or long->str 
conversions.  An obvious example in the core is the tedious 
encode_long() and decode_long() functions in pickle.py.  
Note that a pickle.encode_long() workalike doesn't know in 
advance how many bytes it needs, which would make using 
struct a PITA for that particular use case.  If a proposal isn't 
convenient for taking over existing conversions of this nature, 
that counts against it.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-15 19:31

Message:
Logged In: YES 
user_id=341410

And as I stated, the 's' format character is also a
"variable lengthed type".

It just so happens that in most use cases I've had and
observed for both the 's' format AND proposed 'g' format,
the type size, is in fact, fixed at 'compile' time.  It also
happens that for the 'g' format, this fixed size is not in
the set {1,2,4,8}, which are not limitations for the
pre-existing 's' format.

Please note that the only fundamental difference between the
pre-existing 's' format and the proposed 'g' format, is that
of a quick call to appropriate PyLong_* functions, and a
range check as required by other integer types.


Python is a tool.  Struct is a tool.  By changing the tool
only slightly, we can add flexibility.  The code is already
there, minor glue would make it work, and would make it
convenient for, I believe, more people than binascii.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-09-15 17:26

Message:
Logged In: YES 
user_id=80475

I agree with Michael and Martin that variable length types
do not belong in struct.  The module is about working with
fixed record layouts.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-15 17:17

Message:
Logged In: YES 
user_id=341410

Raymond, from your first post on this topic, it seems as
though you were previously implementing this functionality
in binascii for some particular reason, and it seems as
though it is to be included with binascii in the future,
regardless of the outcome of this particular feature request.

The only reason the binascii solution is better than status
quo, is because a user doesn't need to implement arbitrarily
large integer packing and unpacking themselves.  On the
other hand, it still requires the user make manual
binascii.str_to_long(str_obj) calls in the case of it being
part of a struct, so doesn't gain significantly.

Now, one of the reasons why I requested a format code
addition was because one can (un)pack multiple data types
simultaneously with a single function call via struct.  In
nearly all of the use cases I have for packing and unpacking
large integers, they are a part of other structures.

In the cases where I have been packing and unpacking single
integers, floats, etc., I still use struct because it is has
nearly all of the functionality I need (signed, unsigned,
big endian, little endian, char, short, long, long long,
etc., lacking only arbitrarily large integer packing and
unpacking).

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-09-15 15:08

Message:
Logged In: YES 
user_id=80475

My vote is for binascii functions to parallel hexlify and
unhexlify.   Ideally, it would give the same result as
long(hexlify(s), 16). 





----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-15 11:53

Message:
Logged In: YES 
user_id=341410

As you state, it already supports packing and unpacking a
variable-lengthed type: strings.

In the use cases I've had and seen for (un)packing strings
with struct, it is the most common to define a static format
code, and use that all the time.  That is, you see things
like ">HLLHB25s", that become string constants in a module.
 On the _very rare_ occasion where people want more
flexibility in their types, I have seen both the use of
fixed and variable pascal strings...

def packit(arg1, arg2, arg3, strng):
    return struct.pack(">LHH%ip"%len(strng), arg1, arg2,
arg3, strng)

I would not expect any pascal-string-like packing of a large
integer, though it is possible.  I do expect that most
people have similar use cases as I, and would pre-define
their struct formatting code.  In the case of other similar
requests (long to string, string to long via a base256
representation, etc.) for use in cryptography, I expect that
the regularity of structures used in cryptography would
almost certainly result in formatting codes being module
constants.

To sum up, both in the case for the 's' and 'p' format
codes, and the proposed 'g'/'G' formatting codes, the vast
majority of use cases pre-define the length of the string
and large integer on a per-structure basis via "25s", "25p",
or "25g".  Rarely are the lengths truely variable in the
case of "%ip"%len(strng).

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2004-09-15 10:03

Message:
Logged In: YES 
user_id=6656

Oops, I see you actually address that (ish).  But I still
feel packing what is an essentially variable length type
using the struct module is a bit strange.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2004-09-15 10:02

Message:
Logged In: YES 
user_id=6656

Josiah, what do you suggest struct.calcsize does with the
format code your proposing?  I think this question
encapsulates why I find this feature request a bit misdirected.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-12 11:55

Message:
Logged In: YES 
user_id=341410

Hexlify and unhexlify make sense for translating strings. 
Nowhere in the binascii module is there any mention of
translating anything to/from integers, longs or floats.  The
only reason (un)hexlify make any sense at all is because we
can get integers from hexlified strings, and get strings
from hexlified integers with relative ease.

I guess the trick with struct, at least with me, is not that
I use it because it translates to/from C types, it is
because it translates to/from types that I find useful.  Its
intersection with C types, as well as Python's intersection
with C types, and my use intersection with the types is a
convenient (but very engineered and understandable) coincidence.

It would be another very convenient (but also engineered
*wink*) coincidence if I didn't have to first extract a
section of data, then translate it, in order to get one
large integer.

In the cases that I would truely find useful, big integers
are a part of what would be called structs in the C world,
and wouldn't require additional processing over what I'm
already doing for other integers and floats.


I was looking around, and it turns out that in 2001, Paul
Rubin requested that one be able to translate to/from
arbitrary bases via the C-level format function.  In that
discussion, Paul made the case that there should be a method
to get arbitrarily long integers to and from strings:
"The struct module doesn't give any way of converting
arbitrary ints (meaning longs) to binary.  Really, it's
needed.  Some people do it with gmpy, but if Python is
going to support longs as a built-in type, one shouldn't
have to resort to 3rd-party modules to read and write
them in binary."

Guido followed up with:
"OK, I believe you.  Can you submit a patch?"

It seems like this was in reference to being able to use
functions in binascii for converting to/from arbitrary
packed binary integer types in base 256
(http://sourceforge.net/tracker/?func=detail&atid=105470&aid=465045&group_id=5470
if you are interested).  That request seems to have died
because Paul dropped the ball.

Me, I would prefer struct to binascii, if only because the
code for doing this is already waiting to be used in struct,
and because you can pull multiple objects from a single
packed binary string, rather than one object per call.  This
would seemingly also satisfy complaints of being able to
translate to/from base 256 for arbitrarily large integers.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-09-11 20:40

Message:
Logged In: YES 
user_id=31435

binascii makes sense because that's where the hexlify and 
unhexlify functions live, which are small conceptual steps 
away from what's needed here.

Methods on numbers make sense too, and only seem strange 
because so few are clearly visible now (although, e.g., there 
are lots of them already, like number.__abs__ and 
number.__add__).

The struct module makes sense too, although it would be 
darned ugly to document a refusal to accept the new codes 
in "native" mode; and struct has a high learning curve; and 
struct obviously never intended to support types that aren't 
supplied directly by C compilers (the "Pascal string" code 
seems weird now, but not at the time).

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-10 16:18

Message:
Logged In: YES 
user_id=341410

(sorry it took me a few days to get back to you, I am on a
contract deadline crunch...just taking a break now)

The *HTTPServer heirarchy is interesting in its own right,
but really, each piece in the heirarchy adds functionality.
 A similar thing can be said of asyncore and all the modules
that derive from it (asynchat, *HTTPServer, *XMLRPCServer,
smtpd, etc.).

In this case, since the struct module is already in C and
the functions are not subclassable, creating another module
that parses strings and sends pieces off to struct for
actual decoding seems like a waste of a module, especially
when the change is so minor.  Now, binascii is being used in
such a fashion by uu and binhex, but that is because
binascii is the data processing component, where uu and
binhex make a 'pretty' interface.  Struct doesn't need a
pretty interface, it is already pretty.  Though as I have
said before, I think it could use this small addition.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2004-09-08 15:53

Message:
Logged In: YES 
user_id=21627

Since you were asking: it is quite common that modules refer
to related functionality. For example, BaseHTTPServer refers
to SimpleHTTPServer and CGIHTTPServer. One might expect that
a HTTP server also supports files and does CGI - but not
this one; go elsewhere. Likewise, module binascii refers to
modules uu and binhex. The math documentation points out
that it does not support complex numbers, and that cmath is
needed. The audioop documentation gives the function
echocancel in the documentation, instead of implementing it.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-08 15:38

Message:
Logged In: YES 
user_id=341410

Martin, I was typing as you submitted your most recent comment.

I am honestly shocked that you would suggest that longs
should gain a method for encoding themselves as binary
strings.  Such a thing would then suggest that standard ints
and floats also gain such methods.  It would also imply that
since one can go to strings, one should equivalently be able
to come from strings via equivalent methods.  Goodness,
int.tostring(width) and int.fromstring(str)?  But what about
endianness?

Looks like a big can of worms to me.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-08 15:03

Message:
Logged In: YES 
user_id=341410

Structures (aka C Structs) can contain arbitrarily large or
small numbers of basic types inside them.  As such, 'single long
values' are still a valid use.  I use struct for packing and
unpacking of single items (8,4,2 byte integers, 1 byte
integers are faster served via chr and ord) when necessary
(because it is the most convenient), as well as a current
contract where it is not uncommon to be packing and
unpacking 256 byte structs.

Those large structs contains various 1,2,4 and 8 byte
integers, as well as a handful of 16 and 20 byte integers
(which I must manually shift and mask during packing and
unpacking).  I'm a big boy, and can do it, but that doesn't
mean that such functionality should be left out of Python.

As for 'document the approach of going through hex inside
the documentation of the struct module', I am curious about
whether other modules do the same thing, that is to tell
users "this functionality conceptually fits here X%, which
is why it is documented here, but because it does not fit
100%, here is how you can do the same thing, which will
likely look like a strange hack, require slicing potentially
large strings, and be significantly slower than if we had
just added the functionality, but here you go anyways."

Now, I don't /need/ the feature, but I believe myself and
others would find it useful.  I also don't /require/ it be
in struct, but no other modules offer equivalent
functionality; Pickle and Marshal are Python-only, binascii
(and bin2hex) are for converting between binary and ascii
representations for transferring over non-8-bit channels
(email, web, etc.), and no other module even comes close to
offering a similar bit of "packs various types into a binary
format, the same way C would" as struct.

If anyone has a better place for it, I'm all ears (or eyes).

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2004-09-08 14:45

Message:
Logged In: YES 
user_id=21627

I would think that

def long_as_bytes(lvalue, width):
    fmt = '%%.%dx' % (2*width)
    return unhexlify(fmt % (lvalue & ((1L<<8*width)-1)))

is short enough for a recipe to not really make a C function
necessary for that feature.

However, if they are going to be provided somewhere, I would
suggest that static methods on the long type might be the
right place.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-09-08 12:30

Message:
Logged In: YES 
user_id=80475

The idea is to expose the _PyLong_FromByteArray() and
_PyLong_AsByteArray() functions.  While long(hexlify(b),16)
is doable for bin2long, going the other way is not so simple.

I agree that these are not struct related.  Originally, I
proposed the binascii module because one of the operations
is so similar to hexlify().

As there other suggestions?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2004-09-08 12:15

Message:
Logged In: YES 
user_id=21627

Apparently, the point of this request is that the method for
converting long ints to binary should be "easily found in
documentation". And also apparently, the submitter thinks
that the struct module would be the place where people look.

Now, that allows for a simple solution: document the
approach of going through hex inside the documentation of
the struct module.

There is one other reason (beyond being primarily for C
APIs) why such a feature should *not* be in the struct
module: The struct module, most naturally, is about
structures. However, I understand that the intended usage of
this feature would not be structures, but single long
values. Therefore, I consider it counter-intuitive to extend
struct for that.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-09-06 17:02

Message:
Logged In: YES 
user_id=80475

Okay, submit a patch with docs and unittests.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2004-09-06 16:44

Message:
Logged In: YES 
user_id=341410

As I provide in the feature request, there is already a
method for translating string <-> long.

The problem with current methods for converting between
large integers and strings is that they do not lend
themselves to generally being understandable or to being
documented.

The struct module already provides two appropriate functions
for handling packed binary data, a place for documenting
functions involving packing and unpacking binary data, and
whose implementation seems to be simple enough (one more
format character, much of which borrowed from 's' character,
and a call to _PyLong_FromByteArray seems to be sufficient).

As for the binascii module, many of the functions listed
seem like they should be wrapped into the encode/decode
string methods, hexlify already being so in str.encode('hex').

To me, just being able to translate doesn't seem sufficient
(we already can translate), but being able to do it well,
have it documented well, and placed in a location that is
obvious, fast and optimized for these kinds of things seems
to be the right thing.

>From what I can tell, the only reason why struct doesn't
already have an equivalent format character to the proposed
'g' and 'G', is because the module was created to handle
packed C structs and seemingly "nothing else".  Considering
there doesn't seem to be any other reasonable or easily
documentable location for placing equivalent functionality
(both packing and unpacking), I am of the opinion that
restricting the packing and unpacking to C types in the
struct module (when there are other useful types) is overkill.

As I said, I will provide an implementation if desired.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-09-06 15:34

Message:
Logged In: YES 
user_id=80475

FWIW, I'm working  str/long conversion functions for the
binascii module.  Will that suit your needs?

The tolong function is equivalent to:
    long(hexlify(b), 16)

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470