At the moment, the array module of the standard library allows to
create arrays of different numeric types and to initialize them from
an iterable (eg, another array).
What's missing is the possiblity to specify the final size of the
array (number of items), especially for large arrays.
I'm thinking of suffix arrays (a text indexing data structure) for
large texts, eg the human genome and its reverse complement (about 6
billion characters from the alphabet ACGT).
The suffix array is a long int array of the same size (8 bytes per
number, so it occupies about 48 GB memory).
At the moment I am extending an array in chunks of several million
items at a time at a time, which is slow and not elegant.
The function below also initializes each item in the array to a given
value (0 by default).
Is there a reason why there the array.array constructor does not allow
to simply specify the number of items that should be allocated? (I do
not really care about the contents.)
Would this be a worthwhile addition to / modification of the array module?
My suggestions is to modify array generation in such a way that you
could pass an iterator (as now) as second argument, but if you pass a
single integer value, it should be treated as the number of items to
allocate.
Here is my current workaround (which is slow):
def filled_array(typecode, n, value=0, bsize=(1<<22)):
"""returns a new array with given typecode
(eg, "l" for long int, as in the array module)
with n entries, initialized to the given value (default 0)
"""
a = array.array(typecode, [value]*bsize)
x = array.array(typecode)
r = n
while r >= bsize:
x.extend(a)
r -= bsize
x.extend([value]*r)
return x
Dear all,
I guess this is so obvious that someone must have suggested it before:
in list comprehensions you can currently exclude items based on the if
conditional, e.g.:
[n for n in range(1,1000) if n % 4 == 0]
Why not extend this filtering by allowing a while statement in addition to
if, as in:
[n for n in range(1,1000) while n < 400]
Trivial effect, I agree, in this example since you could achieve the same by
using range(1,400), but I hope you get the point.
This intuitively understandable extension would provide a big speed-up for
sorted lists where processing all the input is unnecessary.
Consider this:
some_names=["Adam", "Andrew", "Arthur", "Bob", "Caroline","Lancelot"] #
a sorted list of names
[n for n in some_names if n.startswith("A")]
# certainly gives a list of all names starting with A, but .
[n for n in some_names while n.startswith("A")]
# would have saved two comparisons
Best,
Wolfgang
While I was implementing JSON-JWS (JSON web signatures), a format
which in Python 3 has to go from bytes > unicode > bytes > unicode
several times in its construction, I notice I wrote a lot of bugs:
"sha256=b'abcdef1234'"
When I meant to say:
"sha256=abcdef1234"
Everything worked perfectly on Python 3 because the verifying code
also generated the sha256=b'abcdef1234' as a comparison. I would have
never noticed at all unless I had tried to verify the Python 3 output
with Python 2.
I know I'm a bad person for not having unit tests capable enough to
catch this bug, a bug I wrote repeatedly in each layer of the bytes >
unicode > bytes > unicode dance, and that there is no excuse for being
confused at any time about the type of a variable, but I'm not willing
to reform.
Instead, I would like a new string formatting operator tentatively
called 'notbytes': "sha256=%notbytes" % (b'abcdef1234'). It gives the
same error as 'sha256='+b'abc1234' would: TypeError: Can't convert
'bytes' object to str implictly
This is similar to another proposal:
http://mail.python.org/pipermail/python-3000/2008-January/011798.html
Anyway, I was using ast.literal_eval and attempted to use frozenset({...})
as a key in a dictionary, which failed, because frozenset isn't a literal
(though putting frozenset in the environment would be a security risk). I
am currently working around this with tuples, but I'd like a literal for
representing frozensets as well. I also use frozensets elsewhere in the
code in ways similar to Raymond's original suggestion.
Perhaps something like f{...} for declaring frozenset( comprehension)?
literals?
Just an idea of usability fix for Python 3.
hexdump module (function or bytes method is better) as simple, easy
and intuitive way for dumping binary data when writing programs in
Python.
hexdump(bytes) - produce human readable dump of binary data,
byte-by-byte representation, separated by space, 16-byte rows
Rationale:
1. Debug.
Generic binary data can't be output to console. A separate helper
is needed to print, log or store its value in human readable format in
database. This takes time.
2. Usability.
binascii is ugly: name is not intuitive any more, there are a lot
of functions, and it is not clear how it relates to unicode.
3. Serialization.
It is convenient to have format that can be displayed in a text
editor. Simple tools encourage people to use them.
Practical example:
>>> print(b)
� � � �� �� � �� �� �
� � �
>>> b
'\xe6\xb0\x08\x04\xe7\x9e\x08\x04\xe7\xbc\x08\x04\xe7\xd5\x08\x04\xe7\xe4\x08\x04\xe6\xb0\x08\x04\xe7\xf0\x08\x04\xe7\xff\x08\x04\xe8\x0b\x08\x04\xe8\x1a\x08\x04\xe6\xb0\x08\x04\xe6\xb0\x08\x04'
>>> print(binascii.hexlify(data))
e6b00804e79e0804e7bc0804e7d50804e7e40804e6b00804e7f00804e7ff0804e80b0804e81a0804e6b00804e6b00804
>>>
>>> data = hexdump(b)
>>> print(data)
E6 B0 08 04 E7 9E 08 04 E7 BC 08 04 E7 D5 08 04
E7 E4 08 04 E6 B0 08 04 E7 F0 08 04 E7 FF 08 04
E8 0B 08 04 E8 1A 08 04 E6 B0 08 04 E6 B0 08 04
>>>
>>> # achieving the same output with binascii is overcomplicated
>>> data_lines = [binascii.hexlify(b)[i:min(i+32, len(binascii.hexlify(b)))] for i in xrange(0, len(binascii.hexlify(b)), 32)]
>>> data_lines = [' '.join(l[i:min(i+2, len(l))] for i in xrange(0, len(l), 2)).upper() for l in data_lines]
>>> print('\n'.join(data_lines))
E6 B0 08 04 E7 9E 08 04 E7 BC 08 04 E7 D5 08 04
E7 E4 08 04 E6 B0 08 04 E7 F0 08 04 E7 FF 08 04
E8 0B 08 04 E8 1A 08 04 E6 B0 08 04 E6 B0 08 04
On the other side, getting rather useless binascii output from
hexdump() is quite trivial:
>>> data.replace(' ','').replace('\n','').lower()
'e6b00804e79e0804e7bc0804e7d50804e7e40804e6b00804e7f00804e7ff0804e80b0804e81a0804e6b00804e6b00804'
But more practical, for example, would be counting offset from hexdump:
>>> print( ''.join( '%05x: %s\n' % (i*16,l) for i,l in enumerate(hexdump(b).split('\n'))))
Etc.
Conclusion:
By providing better building blocks on basic level Python will become
a better tool for more useful tasks.
References:
[1] http://stackoverflow.com/questions/2340319/python-3-1-1-string-to-hex
[2] http://en.wikipedia.org/wiki/Hex_dump
--
anatoly t.
Hi,
Is it interesting to know if people expect -h to work as a --help
equivalent by default?
As it directly affects best practices of using Python in command line, can
we run such poll on python.org? I thought that this stuff is obvious, but
it appears that it is not, so I'd like to see this usability study - it
will be interesting for the Python community to know itself better.
--
anatoly t.
Greetings,
The usual way of resolving configuration is command line -> environment ->
default.
Currently argparse supports only command line -> default, I'd like to
suggest an optional "env" keyword to add_argument that will also resolve
from environment. (And also optional env dictionary to the ArgumentParser
__init__ method [or to parse_args], which will default to os.environ).
Example:
[spam.py]
parser = ArgumentParser()
parser.add_argument('--spam', env='SPAM', default=7)
args = parser.parse_args()
print(args.spam)
./spam.py -> 7
./spam.py --spam=12 -> 12
SPAM=9 ./spam.py -> 9
SPAM=9 ./spam.py --spam=12 -> 12
What do you think?
--
Miki
Just wanted to put out some feelers for the feasibility of these two
features:
* have the **kwargs param be an OrderedDict rather than a dict
* have class definitions happen relative to an OrderedDict by default
rather than a dict, and still overridable by a metaclass's __prepare__().
Both of these will need OrderedDict in C, which is getting close (issue
#16991).
-eric
Dear all,
what do you think about allowing the use of operators on ranges?
I thought of things like:
a = range(1,10)
b = range(5,12)
intersect = a & b # equivalent to intersect = range(5,10)
merge = a | b # equivalent to merge = range(1,12)
to make this work in more complex cases, like:
a = range(1,10)
b = range(12,15)
merge = a | b # equivalent to sth. like range(range(1,10),range(12,15))
maybe range nesting would be desirable, but the full sequence of values of such
a nested range could still be calculated on demand.
Such syntax would allow for the generation of more complex sequences, and, as a
side-effect, ranges could also be checked for overlap easily:
if a & b:
do_something()
this whole idea is reminiscent, of course, of what's implemented for sets
already, so like there you could think of
intersect = a & b as shorthand for intersect = a.intersection(b)
Opinions?
So when putting together my enum implementation I came up against something
I've run into a couple of times in a few different situations, and I
thought warranted some discussion as its own proposal: There are times
where it would be extremely convenient if there was a way when unpacking an
object (using the "a, b, c = foo" notation) for that object to know how
many values it needed to provide.
I would like to propose the following:
When the unpack notation is used, the interpreter will automatically look
for an __unpack__ method on the rvalue, and if it exists will call it as:
rvalue.__unpack__(min_count, max_count)
... and use the returned value as the iterable to unpack into the variables
instead of the original object. (If no __unpack__ method exists, it will
just use the rvalue itself, as it does currently)
In the case of simple unpacking ("a, b, c = foo"), min_count and max_count
will be the same value (3). In the case of extended unpacking ("a, b, c,
*d = foo"), min_count would be the smallest length that will not cause an
unpacking exception (3, in this case), and max_count would be None.
By effectively separating the notion of "unpackable" from "iterable", this
would allow for a few useful things:
1. It would allow otherwise non-iterable objects to support the unpack
notation if it actually makes sense to do so
2. It would allow for "smart" unpackable objects (such as __ in my enum
example), which would "just work" no matter how many values they're
required to produce.
3. It would allow (properly designed) infinite (or
unknown-possibly-large-size) iterators to be unpackable into a finite
sequence of variables if desired (which can only be done with extra
special-case code currently)
4. It could make backwards-compatibility easier for APIs which return
iterables (i.e. if you want to change a function that used to return a
3-tuple to return a 5-tuple, instead of inherently breaking all existing
code that chose to unpack the return value (assuming only 3 items would
ever be returned), you could return an unpackable object which will work
correctly with both old and new code)
Thoughts?
--Alex
