[pypy-commit] pypy numpy-multidim: merge default - tests still broken, working on it
fijal
noreply at buildbot.pypy.org
Fri Oct 28 17:20:51 CEST 2011
Author: Maciej Fijalkowski <fijall at gmail.com>
Branch: numpy-multidim
Changeset: r48566:64aadf44e926
Date: 2011-10-28 17:18 +0200
http://bitbucket.org/pypy/pypy/changeset/64aadf44e926/
Log: merge default - tests still broken, working on it
diff --git a/lib-python/modified-2.7/heapq.py b/lib-python/modified-2.7/heapq.py
new file mode 100644
--- /dev/null
+++ b/lib-python/modified-2.7/heapq.py
@@ -0,0 +1,442 @@
+# -*- coding: latin-1 -*-
+
+"""Heap queue algorithm (a.k.a. priority queue).
+
+Heaps are arrays for which a[k] <= a[2*k+1] and a[k] <= a[2*k+2] for
+all k, counting elements from 0. For the sake of comparison,
+non-existing elements are considered to be infinite. The interesting
+property of a heap is that a[0] is always its smallest element.
+
+Usage:
+
+heap = [] # creates an empty heap
+heappush(heap, item) # pushes a new item on the heap
+item = heappop(heap) # pops the smallest item from the heap
+item = heap[0] # smallest item on the heap without popping it
+heapify(x) # transforms list into a heap, in-place, in linear time
+item = heapreplace(heap, item) # pops and returns smallest item, and adds
+ # new item; the heap size is unchanged
+
+Our API differs from textbook heap algorithms as follows:
+
+- We use 0-based indexing. This makes the relationship between the
+ index for a node and the indexes for its children slightly less
+ obvious, but is more suitable since Python uses 0-based indexing.
+
+- Our heappop() method returns the smallest item, not the largest.
+
+These two make it possible to view the heap as a regular Python list
+without surprises: heap[0] is the smallest item, and heap.sort()
+maintains the heap invariant!
+"""
+
+# Original code by Kevin O'Connor, augmented by Tim Peters and Raymond Hettinger
+
+__about__ = """Heap queues
+
+[explanation by Fran�ois Pinard]
+
+Heaps are arrays for which a[k] <= a[2*k+1] and a[k] <= a[2*k+2] for
+all k, counting elements from 0. For the sake of comparison,
+non-existing elements are considered to be infinite. The interesting
+property of a heap is that a[0] is always its smallest element.
+
+The strange invariant above is meant to be an efficient memory
+representation for a tournament. The numbers below are `k', not a[k]:
+
+ 0
+
+ 1 2
+
+ 3 4 5 6
+
+ 7 8 9 10 11 12 13 14
+
+ 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
+
+
+In the tree above, each cell `k' is topping `2*k+1' and `2*k+2'. In
+an usual binary tournament we see in sports, each cell is the winner
+over the two cells it tops, and we can trace the winner down the tree
+to see all opponents s/he had. However, in many computer applications
+of such tournaments, we do not need to trace the history of a winner.
+To be more memory efficient, when a winner is promoted, we try to
+replace it by something else at a lower level, and the rule becomes
+that a cell and the two cells it tops contain three different items,
+but the top cell "wins" over the two topped cells.
+
+If this heap invariant is protected at all time, index 0 is clearly
+the overall winner. The simplest algorithmic way to remove it and
+find the "next" winner is to move some loser (let's say cell 30 in the
+diagram above) into the 0 position, and then percolate this new 0 down
+the tree, exchanging values, until the invariant is re-established.
+This is clearly logarithmic on the total number of items in the tree.
+By iterating over all items, you get an O(n ln n) sort.
+
+A nice feature of this sort is that you can efficiently insert new
+items while the sort is going on, provided that the inserted items are
+not "better" than the last 0'th element you extracted. This is
+especially useful in simulation contexts, where the tree holds all
+incoming events, and the "win" condition means the smallest scheduled
+time. When an event schedule other events for execution, they are
+scheduled into the future, so they can easily go into the heap. So, a
+heap is a good structure for implementing schedulers (this is what I
+used for my MIDI sequencer :-).
+
+Various structures for implementing schedulers have been extensively
+studied, and heaps are good for this, as they are reasonably speedy,
+the speed is almost constant, and the worst case is not much different
+than the average case. However, there are other representations which
+are more efficient overall, yet the worst cases might be terrible.
+
+Heaps are also very useful in big disk sorts. You most probably all
+know that a big sort implies producing "runs" (which are pre-sorted
+sequences, which size is usually related to the amount of CPU memory),
+followed by a merging passes for these runs, which merging is often
+very cleverly organised[1]. It is very important that the initial
+sort produces the longest runs possible. Tournaments are a good way
+to that. If, using all the memory available to hold a tournament, you
+replace and percolate items that happen to fit the current run, you'll
+produce runs which are twice the size of the memory for random input,
+and much better for input fuzzily ordered.
+
+Moreover, if you output the 0'th item on disk and get an input which
+may not fit in the current tournament (because the value "wins" over
+the last output value), it cannot fit in the heap, so the size of the
+heap decreases. The freed memory could be cleverly reused immediately
+for progressively building a second heap, which grows at exactly the
+same rate the first heap is melting. When the first heap completely
+vanishes, you switch heaps and start a new run. Clever and quite
+effective!
+
+In a word, heaps are useful memory structures to know. I use them in
+a few applications, and I think it is good to keep a `heap' module
+around. :-)
+
+--------------------
+[1] The disk balancing algorithms which are current, nowadays, are
+more annoying than clever, and this is a consequence of the seeking
+capabilities of the disks. On devices which cannot seek, like big
+tape drives, the story was quite different, and one had to be very
+clever to ensure (far in advance) that each tape movement will be the
+most effective possible (that is, will best participate at
+"progressing" the merge). Some tapes were even able to read
+backwards, and this was also used to avoid the rewinding time.
+Believe me, real good tape sorts were quite spectacular to watch!
+From all times, sorting has always been a Great Art! :-)
+"""
+
+__all__ = ['heappush', 'heappop', 'heapify', 'heapreplace', 'merge',
+ 'nlargest', 'nsmallest', 'heappushpop']
+
+from itertools import islice, repeat, count, imap, izip, tee, chain
+from operator import itemgetter
+import bisect
+
+def heappush(heap, item):
+ """Push item onto heap, maintaining the heap invariant."""
+ heap.append(item)
+ _siftdown(heap, 0, len(heap)-1)
+
+def heappop(heap):
+ """Pop the smallest item off the heap, maintaining the heap invariant."""
+ lastelt = heap.pop() # raises appropriate IndexError if heap is empty
+ if heap:
+ returnitem = heap[0]
+ heap[0] = lastelt
+ _siftup(heap, 0)
+ else:
+ returnitem = lastelt
+ return returnitem
+
+def heapreplace(heap, item):
+ """Pop and return the current smallest value, and add the new item.
+
+ This is more efficient than heappop() followed by heappush(), and can be
+ more appropriate when using a fixed-size heap. Note that the value
+ returned may be larger than item! That constrains reasonable uses of
+ this routine unless written as part of a conditional replacement:
+
+ if item > heap[0]:
+ item = heapreplace(heap, item)
+ """
+ returnitem = heap[0] # raises appropriate IndexError if heap is empty
+ heap[0] = item
+ _siftup(heap, 0)
+ return returnitem
+
+def heappushpop(heap, item):
+ """Fast version of a heappush followed by a heappop."""
+ if heap and heap[0] < item:
+ item, heap[0] = heap[0], item
+ _siftup(heap, 0)
+ return item
+
+def heapify(x):
+ """Transform list into a heap, in-place, in O(len(heap)) time."""
+ n = len(x)
+ # Transform bottom-up. The largest index there's any point to looking at
+ # is the largest with a child index in-range, so must have 2*i + 1 < n,
+ # or i < (n-1)/2. If n is even = 2*j, this is (2*j-1)/2 = j-1/2 so
+ # j-1 is the largest, which is n//2 - 1. If n is odd = 2*j+1, this is
+ # (2*j+1-1)/2 = j so j-1 is the largest, and that's again n//2-1.
+ for i in reversed(xrange(n//2)):
+ _siftup(x, i)
+
+def nlargest(n, iterable):
+ """Find the n largest elements in a dataset.
+
+ Equivalent to: sorted(iterable, reverse=True)[:n]
+ """
+ if n < 0: # for consistency with the c impl
+ return []
+ it = iter(iterable)
+ result = list(islice(it, n))
+ if not result:
+ return result
+ heapify(result)
+ _heappushpop = heappushpop
+ for elem in it:
+ _heappushpop(result, elem)
+ result.sort(reverse=True)
+ return result
+
+def nsmallest(n, iterable):
+ """Find the n smallest elements in a dataset.
+
+ Equivalent to: sorted(iterable)[:n]
+ """
+ if n < 0: # for consistency with the c impl
+ return []
+ if hasattr(iterable, '__len__') and n * 10 <= len(iterable):
+ # For smaller values of n, the bisect method is faster than a minheap.
+ # It is also memory efficient, consuming only n elements of space.
+ it = iter(iterable)
+ result = sorted(islice(it, 0, n))
+ if not result:
+ return result
+ insort = bisect.insort
+ pop = result.pop
+ los = result[-1] # los --> Largest of the nsmallest
+ for elem in it:
+ if los <= elem:
+ continue
+ insort(result, elem)
+ pop()
+ los = result[-1]
+ return result
+ # An alternative approach manifests the whole iterable in memory but
+ # saves comparisons by heapifying all at once. Also, saves time
+ # over bisect.insort() which has O(n) data movement time for every
+ # insertion. Finding the n smallest of an m length iterable requires
+ # O(m) + O(n log m) comparisons.
+ h = list(iterable)
+ heapify(h)
+ return map(heappop, repeat(h, min(n, len(h))))
+
+# 'heap' is a heap at all indices >= startpos, except possibly for pos. pos
+# is the index of a leaf with a possibly out-of-order value. Restore the
+# heap invariant.
+def _siftdown(heap, startpos, pos):
+ newitem = heap[pos]
+ # Follow the path to the root, moving parents down until finding a place
+ # newitem fits.
+ while pos > startpos:
+ parentpos = (pos - 1) >> 1
+ parent = heap[parentpos]
+ if newitem < parent:
+ heap[pos] = parent
+ pos = parentpos
+ continue
+ break
+ heap[pos] = newitem
+
+# The child indices of heap index pos are already heaps, and we want to make
+# a heap at index pos too. We do this by bubbling the smaller child of
+# pos up (and so on with that child's children, etc) until hitting a leaf,
+# then using _siftdown to move the oddball originally at index pos into place.
+#
+# We *could* break out of the loop as soon as we find a pos where newitem <=
+# both its children, but turns out that's not a good idea, and despite that
+# many books write the algorithm that way. During a heap pop, the last array
+# element is sifted in, and that tends to be large, so that comparing it
+# against values starting from the root usually doesn't pay (= usually doesn't
+# get us out of the loop early). See Knuth, Volume 3, where this is
+# explained and quantified in an exercise.
+#
+# Cutting the # of comparisons is important, since these routines have no
+# way to extract "the priority" from an array element, so that intelligence
+# is likely to be hiding in custom __cmp__ methods, or in array elements
+# storing (priority, record) tuples. Comparisons are thus potentially
+# expensive.
+#
+# On random arrays of length 1000, making this change cut the number of
+# comparisons made by heapify() a little, and those made by exhaustive
+# heappop() a lot, in accord with theory. Here are typical results from 3
+# runs (3 just to demonstrate how small the variance is):
+#
+# Compares needed by heapify Compares needed by 1000 heappops
+# -------------------------- --------------------------------
+# 1837 cut to 1663 14996 cut to 8680
+# 1855 cut to 1659 14966 cut to 8678
+# 1847 cut to 1660 15024 cut to 8703
+#
+# Building the heap by using heappush() 1000 times instead required
+# 2198, 2148, and 2219 compares: heapify() is more efficient, when
+# you can use it.
+#
+# The total compares needed by list.sort() on the same lists were 8627,
+# 8627, and 8632 (this should be compared to the sum of heapify() and
+# heappop() compares): list.sort() is (unsurprisingly!) more efficient
+# for sorting.
+
+def _siftup(heap, pos):
+ endpos = len(heap)
+ startpos = pos
+ newitem = heap[pos]
+ # Bubble up the smaller child until hitting a leaf.
+ childpos = 2*pos + 1 # leftmost child position
+ while childpos < endpos:
+ # Set childpos to index of smaller child.
+ rightpos = childpos + 1
+ if rightpos < endpos and not heap[childpos] < heap[rightpos]:
+ childpos = rightpos
+ # Move the smaller child up.
+ heap[pos] = heap[childpos]
+ pos = childpos
+ childpos = 2*pos + 1
+ # The leaf at pos is empty now. Put newitem there, and bubble it up
+ # to its final resting place (by sifting its parents down).
+ heap[pos] = newitem
+ _siftdown(heap, startpos, pos)
+
+# If available, use C implementation
+try:
+ from _heapq import *
+except ImportError:
+ pass
+
+def merge(*iterables):
+ '''Merge multiple sorted inputs into a single sorted output.
+
+ Similar to sorted(itertools.chain(*iterables)) but returns a generator,
+ does not pull the data into memory all at once, and assumes that each of
+ the input streams is already sorted (smallest to largest).
+
+ >>> list(merge([1,3,5,7], [0,2,4,8], [5,10,15,20], [], [25]))
+ [0, 1, 2, 3, 4, 5, 5, 7, 8, 10, 15, 20, 25]
+
+ '''
+ _heappop, _heapreplace, _StopIteration = heappop, heapreplace, StopIteration
+
+ h = []
+ h_append = h.append
+ for itnum, it in enumerate(map(iter, iterables)):
+ try:
+ next = it.next
+ h_append([next(), itnum, next])
+ except _StopIteration:
+ pass
+ heapify(h)
+
+ while 1:
+ try:
+ while 1:
+ v, itnum, next = s = h[0] # raises IndexError when h is empty
+ yield v
+ s[0] = next() # raises StopIteration when exhausted
+ _heapreplace(h, s) # restore heap condition
+ except _StopIteration:
+ _heappop(h) # remove empty iterator
+ except IndexError:
+ return
+
+# Extend the implementations of nsmallest and nlargest to use a key= argument
+_nsmallest = nsmallest
+def nsmallest(n, iterable, key=None):
+ """Find the n smallest elements in a dataset.
+
+ Equivalent to: sorted(iterable, key=key)[:n]
+ """
+ # Short-cut for n==1 is to use min() when len(iterable)>0
+ if n == 1:
+ it = iter(iterable)
+ head = list(islice(it, 1))
+ if not head:
+ return []
+ if key is None:
+ return [min(chain(head, it))]
+ return [min(chain(head, it), key=key)]
+
+ # When n>=size, it's faster to use sort()
+ try:
+ size = len(iterable)
+ except (TypeError, AttributeError):
+ pass
+ else:
+ if n >= size:
+ return sorted(iterable, key=key)[:n]
+
+ # When key is none, use simpler decoration
+ if key is None:
+ it = izip(iterable, count()) # decorate
+ result = _nsmallest(n, it)
+ return map(itemgetter(0), result) # undecorate
+
+ # General case, slowest method
+ in1, in2 = tee(iterable)
+ it = izip(imap(key, in1), count(), in2) # decorate
+ result = _nsmallest(n, it)
+ return map(itemgetter(2), result) # undecorate
+
+_nlargest = nlargest
+def nlargest(n, iterable, key=None):
+ """Find the n largest elements in a dataset.
+
+ Equivalent to: sorted(iterable, key=key, reverse=True)[:n]
+ """
+
+ # Short-cut for n==1 is to use max() when len(iterable)>0
+ if n == 1:
+ it = iter(iterable)
+ head = list(islice(it, 1))
+ if not head:
+ return []
+ if key is None:
+ return [max(chain(head, it))]
+ return [max(chain(head, it), key=key)]
+
+ # When n>=size, it's faster to use sort()
+ try:
+ size = len(iterable)
+ except (TypeError, AttributeError):
+ pass
+ else:
+ if n >= size:
+ return sorted(iterable, key=key, reverse=True)[:n]
+
+ # When key is none, use simpler decoration
+ if key is None:
+ it = izip(iterable, count(0,-1)) # decorate
+ result = _nlargest(n, it)
+ return map(itemgetter(0), result) # undecorate
+
+ # General case, slowest method
+ in1, in2 = tee(iterable)
+ it = izip(imap(key, in1), count(0,-1), in2) # decorate
+ result = _nlargest(n, it)
+ return map(itemgetter(2), result) # undecorate
+
+if __name__ == "__main__":
+ # Simple sanity test
+ heap = []
+ data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
+ for item in data:
+ heappush(heap, item)
+ sort = []
+ while heap:
+ sort.append(heappop(heap))
+ print sort
+
+ import doctest
+ doctest.testmod()
diff --git a/lib-python/modified-2.7/test/test_heapq.py b/lib-python/modified-2.7/test/test_heapq.py
--- a/lib-python/modified-2.7/test/test_heapq.py
+++ b/lib-python/modified-2.7/test/test_heapq.py
@@ -186,6 +186,11 @@
self.assertFalse(sys.modules['heapq'] is self.module)
self.assertTrue(hasattr(self.module.heapify, 'func_code'))
+ def test_islice_protection(self):
+ m = self.module
+ self.assertFalse(m.nsmallest(-1, [1]))
+ self.assertFalse(m.nlargest(-1, [1]))
+
class TestHeapC(TestHeap):
module = c_heapq
diff --git a/pypy/jit/backend/llsupport/descr.py b/pypy/jit/backend/llsupport/descr.py
--- a/pypy/jit/backend/llsupport/descr.py
+++ b/pypy/jit/backend/llsupport/descr.py
@@ -281,6 +281,9 @@
def is_float_field(self):
return self.fielddescr.is_float_field()
+ def sort_key(self):
+ return self.fielddescr.sort_key()
+
def repr_of_descr(self):
return '<InteriorFieldDescr %s>' % self.fielddescr.repr_of_descr()
diff --git a/pypy/jit/backend/x86/assembler.py b/pypy/jit/backend/x86/assembler.py
--- a/pypy/jit/backend/x86/assembler.py
+++ b/pypy/jit/backend/x86/assembler.py
@@ -1596,11 +1596,26 @@
genop_getarrayitem_gc_pure = genop_getarrayitem_gc
genop_getarrayitem_raw = genop_getarrayitem_gc
+ def _get_interiorfield_addr(self, temp_loc, index_loc, itemsize_loc,
+ base_loc, ofs_loc):
+ assert isinstance(itemsize_loc, ImmedLoc)
+ if isinstance(index_loc, ImmedLoc):
+ temp_loc = imm(index_loc.value * itemsize_loc.value)
+ else:
+ # XXX should not use IMUL in most cases
+ assert isinstance(temp_loc, RegLoc)
+ assert isinstance(index_loc, RegLoc)
+ self.mc.IMUL_rri(temp_loc.value, index_loc.value,
+ itemsize_loc.value)
+ assert isinstance(ofs_loc, ImmedLoc)
+ return AddressLoc(base_loc, temp_loc, 0, ofs_loc.value)
+
def genop_getinteriorfield_gc(self, op, arglocs, resloc):
- base_loc, ofs_loc, itemsize_loc, fieldsize_loc, index_loc, sign_loc = arglocs
- # XXX should not use IMUL in most cases
- self.mc.IMUL(index_loc, itemsize_loc)
- src_addr = AddressLoc(base_loc, index_loc, 0, ofs_loc.value)
+ (base_loc, ofs_loc, itemsize_loc, fieldsize_loc,
+ index_loc, sign_loc) = arglocs
+ src_addr = self._get_interiorfield_addr(resloc, index_loc,
+ itemsize_loc, base_loc,
+ ofs_loc)
self.load_from_mem(resloc, src_addr, fieldsize_loc, sign_loc)
@@ -1611,13 +1626,11 @@
self.save_into_mem(dest_addr, value_loc, size_loc)
def genop_discard_setinteriorfield_gc(self, op, arglocs):
- base_loc, ofs_loc, itemsize_loc, fieldsize_loc, index_loc, value_loc = arglocs
- # XXX should not use IMUL in most cases
- if isinstance(index_loc, ImmedLoc):
- index_loc = imm(index_loc.value * itemsize_loc.value)
- else:
- self.mc.IMUL(index_loc, itemsize_loc)
- dest_addr = AddressLoc(base_loc, index_loc, 0, ofs_loc.value)
+ (base_loc, ofs_loc, itemsize_loc, fieldsize_loc,
+ index_loc, temp_loc, value_loc) = arglocs
+ dest_addr = self._get_interiorfield_addr(temp_loc, index_loc,
+ itemsize_loc, base_loc,
+ ofs_loc)
self.save_into_mem(dest_addr, value_loc, fieldsize_loc)
def genop_discard_setarrayitem_gc(self, op, arglocs):
diff --git a/pypy/jit/backend/x86/regalloc.py b/pypy/jit/backend/x86/regalloc.py
--- a/pypy/jit/backend/x86/regalloc.py
+++ b/pypy/jit/backend/x86/regalloc.py
@@ -1046,16 +1046,26 @@
need_lower_byte = True
else:
need_lower_byte = False
- base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
- tempvar = TempBox()
- index_loc = self.rm.force_result_in_reg(tempvar, op.getarg(1), args)
- # we're free to modify index now
- value_loc = self.make_sure_var_in_reg(op.getarg(2), args + [tempvar],
+ box_base, box_index, box_value = args
+ base_loc = self.rm.make_sure_var_in_reg(box_base, args)
+ index_loc = self.rm.make_sure_var_in_reg(box_index, args)
+ value_loc = self.make_sure_var_in_reg(box_value, args,
need_lower_byte=need_lower_byte)
- self.rm.possibly_free_var(tempvar)
- self.possibly_free_vars(args)
+ # If 'index_loc' is not an immediate, then we need a 'temp_loc' that
+ # is a register whose value will be destroyed. It's fine to destroy
+ # the same register as 'index_loc', but not the other ones.
+ self.rm.possibly_free_var(box_index)
+ if not isinstance(index_loc, ImmedLoc):
+ tempvar = TempBox()
+ temp_loc = self.rm.force_allocate_reg(tempvar, [box_base,
+ box_value])
+ self.rm.possibly_free_var(tempvar)
+ else:
+ temp_loc = None
+ self.rm.possibly_free_var(box_base)
+ self.possibly_free_var(box_value)
self.PerformDiscard(op, [base_loc, ofs, itemsize, fieldsize,
- index_loc, value_loc])
+ index_loc, temp_loc, value_loc])
def consider_strsetitem(self, op):
args = op.getarglist()
@@ -1126,13 +1136,14 @@
else:
sign_loc = imm0
args = op.getarglist()
- tmpvar = TempBox()
base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
- index_loc = self.rm.force_result_in_reg(tmpvar, op.getarg(1),
- args)
- self.rm.possibly_free_vars_for_op(op)
- self.rm.possibly_free_var(tmpvar)
- result_loc = self.force_allocate_reg(op.result)
+ index_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
+ # 'base' and 'index' are put in two registers (or one if 'index'
+ # is an immediate). 'result' can be in the same register as
+ # 'index' but must be in a different register than 'base'.
+ self.rm.possibly_free_var(op.getarg(1))
+ result_loc = self.force_allocate_reg(op.result, [op.getarg(0)])
+ self.rm.possibly_free_var(op.getarg(0))
self.Perform(op, [base_loc, ofs, itemsize, fieldsize,
index_loc, sign_loc], result_loc)
diff --git a/pypy/jit/codewriter/jtransform.py b/pypy/jit/codewriter/jtransform.py
--- a/pypy/jit/codewriter/jtransform.py
+++ b/pypy/jit/codewriter/jtransform.py
@@ -844,6 +844,10 @@
if self._is_gc(op.args[0]):
return op
+ def rewrite_op_cast_opaque_ptr(self, op):
+ # None causes the result of this op to get aliased to op.args[0]
+ return [SpaceOperation('mark_opaque_ptr', op.args, None), None]
+
def rewrite_op_force_cast(self, op):
v_arg = op.args[0]
v_result = op.result
diff --git a/pypy/jit/codewriter/test/test_jtransform.py b/pypy/jit/codewriter/test/test_jtransform.py
--- a/pypy/jit/codewriter/test/test_jtransform.py
+++ b/pypy/jit/codewriter/test/test_jtransform.py
@@ -1128,3 +1128,16 @@
varoftype(lltype.Signed))
tr = Transformer(None, None)
raises(NotImplementedError, tr.rewrite_operation, op)
+
+def test_cast_opaque_ptr():
+ S = lltype.GcStruct("S", ("x", lltype.Signed))
+ v1 = varoftype(lltype.Ptr(S))
+ v2 = varoftype(lltype.Ptr(rclass.OBJECT))
+
+ op = SpaceOperation('cast_opaque_ptr', [v1], v2)
+ tr = Transformer()
+ [op1, op2] = tr.rewrite_operation(op)
+ assert op1.opname == 'mark_opaque_ptr'
+ assert op1.args == [v1]
+ assert op1.result is None
+ assert op2 is None
\ No newline at end of file
diff --git a/pypy/jit/metainterp/blackhole.py b/pypy/jit/metainterp/blackhole.py
--- a/pypy/jit/metainterp/blackhole.py
+++ b/pypy/jit/metainterp/blackhole.py
@@ -505,9 +505,6 @@
@arguments("r", "r", returns="i")
def bhimpl_instance_ptr_ne(a, b):
return a != b
- @arguments("r", returns="r")
- def bhimpl_cast_opaque_ptr(a):
- return a
@arguments("r", returns="i")
def bhimpl_cast_ptr_to_int(a):
i = lltype.cast_ptr_to_int(a)
@@ -518,6 +515,10 @@
ll_assert((i & 1) == 1, "bhimpl_cast_int_to_ptr: not an odd int")
return lltype.cast_int_to_ptr(llmemory.GCREF, i)
+ @arguments("r")
+ def bhimpl_mark_opaque_ptr(a):
+ pass
+
@arguments("i", returns="i")
def bhimpl_int_copy(a):
return a
diff --git a/pypy/jit/metainterp/heapcache.py b/pypy/jit/metainterp/heapcache.py
--- a/pypy/jit/metainterp/heapcache.py
+++ b/pypy/jit/metainterp/heapcache.py
@@ -34,7 +34,6 @@
self.clear_caches(opnum, descr, argboxes)
def mark_escaped(self, opnum, argboxes):
- idx = 0
if opnum == rop.SETFIELD_GC:
assert len(argboxes) == 2
box, valuebox = argboxes
@@ -42,8 +41,20 @@
self.dependencies.setdefault(box, []).append(valuebox)
else:
self._escape(valuebox)
- # GETFIELD_GC doesn't escape it's argument
- elif opnum != rop.GETFIELD_GC:
+ elif opnum == rop.SETARRAYITEM_GC:
+ assert len(argboxes) == 3
+ box, indexbox, valuebox = argboxes
+ if self.is_unescaped(box) and self.is_unescaped(valuebox):
+ self.dependencies.setdefault(box, []).append(valuebox)
+ else:
+ self._escape(valuebox)
+ # GETFIELD_GC, MARK_OPAQUE_PTR, PTR_EQ, and PTR_NE don't escape their
+ # arguments
+ elif (opnum != rop.GETFIELD_GC and
+ opnum != rop.MARK_OPAQUE_PTR and
+ opnum != rop.PTR_EQ and
+ opnum != rop.PTR_NE):
+ idx = 0
for box in argboxes:
# setarrayitem_gc don't escape its first argument
if not (idx == 0 and opnum in [rop.SETARRAYITEM_GC]):
@@ -60,13 +71,13 @@
self._escape(dep)
def clear_caches(self, opnum, descr, argboxes):
- if opnum == rop.SETFIELD_GC:
- return
- if opnum == rop.SETARRAYITEM_GC:
- return
- if opnum == rop.SETFIELD_RAW:
- return
- if opnum == rop.SETARRAYITEM_RAW:
+ if (opnum == rop.SETFIELD_GC or
+ opnum == rop.SETARRAYITEM_GC or
+ opnum == rop.SETFIELD_RAW or
+ opnum == rop.SETARRAYITEM_RAW or
+ opnum == rop.SETINTERIORFIELD_GC or
+ opnum == rop.COPYSTRCONTENT or
+ opnum == rop.COPYUNICODECONTENT):
return
if rop._OVF_FIRST <= opnum <= rop._OVF_LAST:
return
@@ -75,9 +86,9 @@
if opnum == rop.CALL or opnum == rop.CALL_LOOPINVARIANT:
effectinfo = descr.get_extra_info()
ef = effectinfo.extraeffect
- if ef == effectinfo.EF_LOOPINVARIANT or \
- ef == effectinfo.EF_ELIDABLE_CANNOT_RAISE or \
- ef == effectinfo.EF_ELIDABLE_CAN_RAISE:
+ if (ef == effectinfo.EF_LOOPINVARIANT or
+ ef == effectinfo.EF_ELIDABLE_CANNOT_RAISE or
+ ef == effectinfo.EF_ELIDABLE_CAN_RAISE):
return
# A special case for ll_arraycopy, because it is so common, and its
# effects are so well defined.
diff --git a/pypy/jit/metainterp/history.py b/pypy/jit/metainterp/history.py
--- a/pypy/jit/metainterp/history.py
+++ b/pypy/jit/metainterp/history.py
@@ -929,6 +929,9 @@
def view(self, **kwds):
pass
+ def clear(self):
+ pass
+
class Stats(object):
"""For tests."""
@@ -943,6 +946,15 @@
self.aborted_keys = []
self.invalidated_token_numbers = set()
+ def clear(self):
+ del self.loops[:]
+ del self.locations[:]
+ del self.aborted_keys[:]
+ self.invalidated_token_numbers.clear()
+ self.compiled_count = 0
+ self.enter_count = 0
+ self.aborted_count = 0
+
def set_history(self, history):
self.operations = history.operations
diff --git a/pypy/jit/metainterp/optimizeopt/optimizer.py b/pypy/jit/metainterp/optimizeopt/optimizer.py
--- a/pypy/jit/metainterp/optimizeopt/optimizer.py
+++ b/pypy/jit/metainterp/optimizeopt/optimizer.py
@@ -209,13 +209,19 @@
def setfield(self, ofs, value):
raise NotImplementedError
+ def getlength(self):
+ raise NotImplementedError
+
def getitem(self, index):
raise NotImplementedError
- def getlength(self):
+ def setitem(self, index, value):
raise NotImplementedError
- def setitem(self, index, value):
+ def getinteriorfield(self, index, ofs, default):
+ raise NotImplementedError
+
+ def setinteriorfield(self, index, ofs, value):
raise NotImplementedError
@@ -283,11 +289,11 @@
return self.optimizer.optpure.has_pure_result(opnum, args, descr)
return False
- def get_pure_result(self, key):
+ def get_pure_result(self, key):
if self.optimizer.optpure:
return self.optimizer.optpure.get_pure_result(key)
return None
-
+
def setup(self):
pass
@@ -524,7 +530,7 @@
def replace_op(self, old_op, new_op):
# XXX: Do we want to cache indexes to prevent search?
- i = len(self._newoperations)
+ i = len(self._newoperations)
while i > 0:
i -= 1
if self._newoperations[i] is old_op:
diff --git a/pypy/jit/metainterp/optimizeopt/rewrite.py b/pypy/jit/metainterp/optimizeopt/rewrite.py
--- a/pypy/jit/metainterp/optimizeopt/rewrite.py
+++ b/pypy/jit/metainterp/optimizeopt/rewrite.py
@@ -465,10 +465,9 @@
args = [op.getarg(0), ConstInt(highest_bit(val))])
self.emit_operation(op)
- def optimize_CAST_OPAQUE_PTR(self, op):
+ def optimize_MARK_OPAQUE_PTR(self, op):
value = self.getvalue(op.getarg(0))
self.optimizer.opaque_pointers[value] = True
- self.make_equal_to(op.result, value)
def optimize_CAST_PTR_TO_INT(self, op):
self.pure(rop.CAST_INT_TO_PTR, [op.result], op.getarg(0))
diff --git a/pypy/jit/metainterp/optimizeopt/simplify.py b/pypy/jit/metainterp/optimizeopt/simplify.py
--- a/pypy/jit/metainterp/optimizeopt/simplify.py
+++ b/pypy/jit/metainterp/optimizeopt/simplify.py
@@ -25,7 +25,8 @@
# but it's a bit hard to implement robustly if heap.py is also run
pass
- optimize_CAST_OPAQUE_PTR = optimize_VIRTUAL_REF
+ def optimize_MARK_OPAQUE_PTR(self, op):
+ pass
dispatch_opt = make_dispatcher_method(OptSimplify, 'optimize_',
diff --git a/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py b/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py
--- a/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py
+++ b/pypy/jit/metainterp/optimizeopt/test/test_optimizebasic.py
@@ -935,7 +935,6 @@
"""
self.optimize_loop(ops, expected)
-
def test_virtual_constant_isnonnull(self):
ops = """
[i0]
@@ -951,6 +950,32 @@
"""
self.optimize_loop(ops, expected)
+ def test_virtual_array_of_struct(self):
+ ops = """
+ [f0, f1, f2, f3]
+ p0 = new_array(2, descr=complexarraydescr)
+ setinteriorfield_gc(p0, 0, f0, descr=complexrealdescr)
+ setinteriorfield_gc(p0, 0, f1, descr=compleximagdescr)
+ setinteriorfield_gc(p0, 1, f2, descr=complexrealdescr)
+ setinteriorfield_gc(p0, 1, f3, descr=compleximagdescr)
+ f4 = getinteriorfield_gc(p0, 0, descr=complexrealdescr)
+ f5 = getinteriorfield_gc(p0, 1, descr=complexrealdescr)
+ f6 = float_mul(f4, f5)
+ f7 = getinteriorfield_gc(p0, 0, descr=compleximagdescr)
+ f8 = getinteriorfield_gc(p0, 1, descr=compleximagdescr)
+ f9 = float_mul(f7, f8)
+ f10 = float_add(f6, f9)
+ finish(f10)
+ """
+ expected = """
+ [f0, f1, f2, f3]
+ f4 = float_mul(f0, f2)
+ f5 = float_mul(f1, f3)
+ f6 = float_add(f4, f5)
+ finish(f6)
+ """
+ self.optimize_loop(ops, expected)
+
def test_nonvirtual_1(self):
ops = """
[i]
@@ -4181,10 +4206,12 @@
class FakeCallInfoCollection:
def callinfo_for_oopspec(self, oopspecindex):
calldescrtype = type(LLtypeMixin.strequaldescr)
+ effectinfotype = type(LLtypeMixin.strequaldescr.get_extra_info())
for value in LLtypeMixin.__dict__.values():
if isinstance(value, calldescrtype):
extra = value.get_extra_info()
- if extra and extra.oopspecindex == oopspecindex:
+ if (extra and isinstance(extra, effectinfotype) and
+ extra.oopspecindex == oopspecindex):
# returns 0 for 'func' in this test
return value, 0
raise AssertionError("not found: oopspecindex=%d" %
diff --git a/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py b/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py
--- a/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py
+++ b/pypy/jit/metainterp/optimizeopt/test/test_optimizeopt.py
@@ -5800,10 +5800,12 @@
class FakeCallInfoCollection:
def callinfo_for_oopspec(self, oopspecindex):
calldescrtype = type(LLtypeMixin.strequaldescr)
+ effectinfotype = type(LLtypeMixin.strequaldescr.get_extra_info())
for value in LLtypeMixin.__dict__.values():
if isinstance(value, calldescrtype):
extra = value.get_extra_info()
- if extra and extra.oopspecindex == oopspecindex:
+ if (extra and isinstance(extra, effectinfotype) and
+ extra.oopspecindex == oopspecindex):
# returns 0 for 'func' in this test
return value, 0
raise AssertionError("not found: oopspecindex=%d" %
diff --git a/pypy/jit/metainterp/optimizeopt/test/test_util.py b/pypy/jit/metainterp/optimizeopt/test/test_util.py
--- a/pypy/jit/metainterp/optimizeopt/test/test_util.py
+++ b/pypy/jit/metainterp/optimizeopt/test/test_util.py
@@ -185,6 +185,18 @@
EffectInfo([], [arraydescr], [], [arraydescr],
oopspecindex=EffectInfo.OS_ARRAYCOPY))
+
+ # array of structs (complex data)
+ complexarray = lltype.GcArray(
+ lltype.Struct("complex",
+ ("real", lltype.Float),
+ ("imag", lltype.Float),
+ )
+ )
+ complexarraydescr = cpu.arraydescrof(complexarray)
+ complexrealdescr = cpu.interiorfielddescrof(complexarray, "real")
+ compleximagdescr = cpu.interiorfielddescrof(complexarray, "imag")
+
for _name, _os in [
('strconcatdescr', 'OS_STR_CONCAT'),
('strslicedescr', 'OS_STR_SLICE'),
@@ -240,7 +252,7 @@
## def get_class_of_box(self, box):
## root = box.getref(ootype.ROOT)
## return ootype.classof(root)
-
+
## cpu = runner.OOtypeCPU(None)
## NODE = ootype.Instance('NODE', ootype.ROOT, {})
## NODE._add_fields({'value': ootype.Signed,
diff --git a/pypy/jit/metainterp/optimizeopt/virtualize.py b/pypy/jit/metainterp/optimizeopt/virtualize.py
--- a/pypy/jit/metainterp/optimizeopt/virtualize.py
+++ b/pypy/jit/metainterp/optimizeopt/virtualize.py
@@ -271,6 +271,69 @@
def _make_virtual(self, modifier):
return modifier.make_varray(self.arraydescr)
+class VArrayStructValue(AbstractVirtualValue):
+ def __init__(self, arraydescr, size, keybox, source_op=None):
+ AbstractVirtualValue.__init__(self, keybox, source_op)
+ self.arraydescr = arraydescr
+ self._items = [{} for _ in xrange(size)]
+
+ def getlength(self):
+ return len(self._items)
+
+ def getinteriorfield(self, index, ofs, default):
+ return self._items[index].get(ofs, default)
+
+ def setinteriorfield(self, index, ofs, itemvalue):
+ assert isinstance(itemvalue, optimizer.OptValue)
+ self._items[index][ofs] = itemvalue
+
+ def _really_force(self, optforce):
+ assert self.source_op is not None
+ if not we_are_translated():
+ self.source_op.name = 'FORCE ' + self.source_op.name
+ optforce.emit_operation(self.source_op)
+ self.box = box = self.source_op.result
+ for index in range(len(self._items)):
+ for descr, value in self._items[index].iteritems():
+ subbox = value.force_box(optforce)
+ op = ResOperation(rop.SETINTERIORFIELD_GC,
+ [box, ConstInt(index), subbox], None, descr=descr
+ )
+ optforce.emit_operation(op)
+
+ def _get_list_of_descrs(self):
+ descrs = []
+ for item in self._items:
+ item_descrs = item.keys()
+ sort_descrs(item_descrs)
+ descrs.append(item_descrs)
+ return descrs
+
+ def get_args_for_fail(self, modifier):
+ if self.box is None and not modifier.already_seen_virtual(self.keybox):
+ itemdescrs = self._get_list_of_descrs()
+ itemboxes = []
+ for i in range(len(self._items)):
+ for descr in itemdescrs[i]:
+ itemboxes.append(self._items[i][descr].get_key_box())
+ modifier.register_virtual_fields(self.keybox, itemboxes)
+ for i in range(len(self._items)):
+ for descr in itemdescrs[i]:
+ self._items[i][descr].get_args_for_fail(modifier)
+
+ def force_at_end_of_preamble(self, already_forced, optforce):
+ if self in already_forced:
+ return self
+ already_forced[self] = self
+ for index in range(len(self._items)):
+ for descr in self._items[index].keys():
+ self._items[index][descr] = self._items[index][descr].force_at_end_of_preamble(already_forced, optforce)
+ return self
+
+ def _make_virtual(self, modifier):
+ return modifier.make_varraystruct(self.arraydescr, self._get_list_of_descrs())
+
+
class OptVirtualize(optimizer.Optimization):
"Virtualize objects until they escape."
@@ -283,8 +346,11 @@
return vvalue
def make_varray(self, arraydescr, size, box, source_op=None):
- constvalue = self.new_const_item(arraydescr)
- vvalue = VArrayValue(arraydescr, constvalue, size, box, source_op)
+ if arraydescr.is_array_of_structs():
+ vvalue = VArrayStructValue(arraydescr, size, box, source_op)
+ else:
+ constvalue = self.new_const_item(arraydescr)
+ vvalue = VArrayValue(arraydescr, constvalue, size, box, source_op)
self.make_equal_to(box, vvalue)
return vvalue
@@ -386,8 +452,7 @@
def optimize_NEW_ARRAY(self, op):
sizebox = self.get_constant_box(op.getarg(0))
- # For now we can't make arrays of structs virtual.
- if sizebox is not None and not op.getdescr().is_array_of_structs():
+ if sizebox is not None:
# if the original 'op' did not have a ConstInt as argument,
# build a new one with the ConstInt argument
if not isinstance(op.getarg(0), ConstInt):
@@ -432,6 +497,34 @@
value.ensure_nonnull()
self.emit_operation(op)
+ def optimize_GETINTERIORFIELD_GC(self, op):
+ value = self.getvalue(op.getarg(0))
+ if value.is_virtual():
+ indexbox = self.get_constant_box(op.getarg(1))
+ if indexbox is not None:
+ descr = op.getdescr()
+ fieldvalue = value.getinteriorfield(
+ indexbox.getint(), descr, None
+ )
+ if fieldvalue is None:
+ fieldvalue = self.new_const(descr)
+ self.make_equal_to(op.result, fieldvalue)
+ return
+ value.ensure_nonnull()
+ self.emit_operation(op)
+
+ def optimize_SETINTERIORFIELD_GC(self, op):
+ value = self.getvalue(op.getarg(0))
+ if value.is_virtual():
+ indexbox = self.get_constant_box(op.getarg(1))
+ if indexbox is not None:
+ value.setinteriorfield(
+ indexbox.getint(), op.getdescr(), self.getvalue(op.getarg(2))
+ )
+ return
+ value.ensure_nonnull()
+ self.emit_operation(op)
+
dispatch_opt = make_dispatcher_method(OptVirtualize, 'optimize_',
default=OptVirtualize.emit_operation)
diff --git a/pypy/jit/metainterp/optimizeopt/virtualstate.py b/pypy/jit/metainterp/optimizeopt/virtualstate.py
--- a/pypy/jit/metainterp/optimizeopt/virtualstate.py
+++ b/pypy/jit/metainterp/optimizeopt/virtualstate.py
@@ -16,7 +16,7 @@
class AbstractVirtualStateInfo(resume.AbstractVirtualInfo):
position = -1
-
+
def generalization_of(self, other, renum, bad):
raise NotImplementedError
@@ -54,7 +54,7 @@
s.debug_print(indent + " ", seen, bad)
else:
debug_print(indent + " ...")
-
+
def debug_header(self, indent):
raise NotImplementedError
@@ -77,13 +77,15 @@
bad[self] = True
bad[other] = True
return False
+
+ assert isinstance(other, AbstractVirtualStructStateInfo)
assert len(self.fielddescrs) == len(self.fieldstate)
assert len(other.fielddescrs) == len(other.fieldstate)
if len(self.fielddescrs) != len(other.fielddescrs):
bad[self] = True
bad[other] = True
return False
-
+
for i in range(len(self.fielddescrs)):
if other.fielddescrs[i] is not self.fielddescrs[i]:
bad[self] = True
@@ -112,8 +114,8 @@
def _enum(self, virtual_state):
for s in self.fieldstate:
s.enum(virtual_state)
-
-
+
+
class VirtualStateInfo(AbstractVirtualStructStateInfo):
def __init__(self, known_class, fielddescrs):
AbstractVirtualStructStateInfo.__init__(self, fielddescrs)
@@ -128,13 +130,13 @@
def debug_header(self, indent):
debug_print(indent + 'VirtualStateInfo(%d):' % self.position)
-
+
class VStructStateInfo(AbstractVirtualStructStateInfo):
def __init__(self, typedescr, fielddescrs):
AbstractVirtualStructStateInfo.__init__(self, fielddescrs)
self.typedescr = typedescr
- def _generalization_of(self, other):
+ def _generalization_of(self, other):
if not isinstance(other, VStructStateInfo):
return False
if self.typedescr is not other.typedescr:
@@ -143,7 +145,7 @@
def debug_header(self, indent):
debug_print(indent + 'VStructStateInfo(%d):' % self.position)
-
+
class VArrayStateInfo(AbstractVirtualStateInfo):
def __init__(self, arraydescr):
self.arraydescr = arraydescr
@@ -157,11 +159,7 @@
bad[other] = True
return False
renum[self.position] = other.position
- if not isinstance(other, VArrayStateInfo):
- bad[self] = True
- bad[other] = True
- return False
- if self.arraydescr is not other.arraydescr:
+ if not self._generalization_of(other):
bad[self] = True
bad[other] = True
return False
@@ -177,6 +175,10 @@
return False
return True
+ def _generalization_of(self, other):
+ return (isinstance(other, VArrayStateInfo) and
+ self.arraydescr is other.arraydescr)
+
def enum_forced_boxes(self, boxes, value, optimizer):
assert isinstance(value, virtualize.VArrayValue)
assert value.is_virtual()
@@ -192,8 +194,75 @@
def debug_header(self, indent):
debug_print(indent + 'VArrayStateInfo(%d):' % self.position)
-
-
+
+class VArrayStructStateInfo(AbstractVirtualStateInfo):
+ def __init__(self, arraydescr, fielddescrs):
+ self.arraydescr = arraydescr
+ self.fielddescrs = fielddescrs
+
+ def generalization_of(self, other, renum, bad):
+ assert self.position != -1
+ if self.position in renum:
+ if renum[self.position] == other.position:
+ return True
+ bad[self] = True
+ bad[other] = True
+ return False
+ renum[self.position] = other.position
+ if not self._generalization_of(other):
+ bad[self] = True
+ bad[other] = True
+ return False
+
+ assert isinstance(other, VArrayStructStateInfo)
+ if len(self.fielddescrs) != len(other.fielddescrs):
+ bad[self] = True
+ bad[other] = True
+ return False
+
+ p = 0
+ for i in range(len(self.fielddescrs)):
+ if len(self.fielddescrs[i]) != len(other.fielddescrs[i]):
+ bad[self] = True
+ bad[other] = True
+ return False
+ for j in range(len(self.fielddescrs[i])):
+ if self.fielddescrs[i][j] is not other.fielddescrs[i][j]:
+ bad[self] = True
+ bad[other] = True
+ return False
+ if not self.fieldstate[p].generalization_of(other.fieldstate[p],
+ renum, bad):
+ bad[self] = True
+ bad[other] = True
+ return False
+ p += 1
+ return True
+
+ def _generalization_of(self, other):
+ return (isinstance(other, VArrayStructStateInfo) and
+ self.arraydescr is other.arraydescr)
+
+ def _enum(self, virtual_state):
+ for s in self.fieldstate:
+ s.enum(virtual_state)
+
+ def enum_forced_boxes(self, boxes, value, optimizer):
+ assert isinstance(value, virtualize.VArrayStructValue)
+ assert value.is_virtual()
+ p = 0
+ for i in range(len(self.fielddescrs)):
+ for j in range(len(self.fielddescrs[i])):
+ v = value._items[i][self.fielddescrs[i][j]]
+ s = self.fieldstate[p]
+ if s.position > self.position:
+ s.enum_forced_boxes(boxes, v, optimizer)
+ p += 1
+
+ def debug_header(self, indent):
+ debug_print(indent + 'VArrayStructStateInfo(%d):' % self.position)
+
+
class NotVirtualStateInfo(AbstractVirtualStateInfo):
def __init__(self, value):
self.known_class = value.known_class
@@ -277,7 +346,7 @@
op = ResOperation(rop.GUARD_CLASS, [box, self.known_class], None)
extra_guards.append(op)
return
-
+
if self.level == LEVEL_NONNULL and \
other.level == LEVEL_UNKNOWN and \
isinstance(box, BoxPtr) and \
@@ -285,7 +354,7 @@
op = ResOperation(rop.GUARD_NONNULL, [box], None)
extra_guards.append(op)
return
-
+
if self.level == LEVEL_UNKNOWN and \
other.level == LEVEL_UNKNOWN and \
isinstance(box, BoxInt) and \
@@ -309,7 +378,7 @@
op = ResOperation(rop.GUARD_TRUE, [res], None)
extra_guards.append(op)
return
-
+
# Remaining cases are probably not interesting
raise InvalidLoop
if self.level == LEVEL_CONSTANT:
@@ -319,7 +388,7 @@
def enum_forced_boxes(self, boxes, value, optimizer):
if self.level == LEVEL_CONSTANT:
return
- assert 0 <= self.position_in_notvirtuals
+ assert 0 <= self.position_in_notvirtuals
boxes[self.position_in_notvirtuals] = value.force_box(optimizer)
def _enum(self, virtual_state):
@@ -348,7 +417,7 @@
lb = ''
if self.lenbound:
lb = ', ' + self.lenbound.bound.__repr__()
-
+
debug_print(indent + mark + 'NotVirtualInfo(%d' % self.position +
', ' + l + ', ' + self.intbound.__repr__() + lb + ')')
@@ -370,7 +439,7 @@
return False
return True
- def generate_guards(self, other, args, cpu, extra_guards):
+ def generate_guards(self, other, args, cpu, extra_guards):
assert len(self.state) == len(other.state) == len(args)
renum = {}
for i in range(len(self.state)):
@@ -393,7 +462,7 @@
inputargs.append(box)
assert None not in inputargs
-
+
return inputargs
def debug_print(self, hdr='', bad=None):
@@ -412,7 +481,7 @@
def register_virtual_fields(self, keybox, fieldboxes):
self.fieldboxes[keybox] = fieldboxes
-
+
def already_seen_virtual(self, keybox):
return keybox in self.fieldboxes
@@ -463,6 +532,9 @@
def make_varray(self, arraydescr):
return VArrayStateInfo(arraydescr)
+ def make_varraystruct(self, arraydescr, fielddescrs):
+ return VArrayStructStateInfo(arraydescr, fielddescrs)
+
class BoxNotProducable(Exception):
pass
@@ -501,12 +573,12 @@
else: # Low priority
lo -= 1
return alts
-
+
def renamed(self, box):
if box in self.rename:
return self.rename[box]
return box
-
+
def add_to_short(self, box, op):
if op:
op = op.clone()
@@ -528,12 +600,12 @@
self.optimizer.make_equal_to(newbox, value)
else:
self.short_boxes[box] = op
-
+
def produce_short_preamble_box(self, box):
if box in self.short_boxes:
- return
+ return
if isinstance(box, Const):
- return
+ return
if box in self.potential_ops:
ops = self.prioritized_alternatives(box)
produced_one = False
@@ -570,7 +642,7 @@
else:
debug_print(logops.repr_of_arg(box) + ': None')
debug_stop('jit-short-boxes')
-
+
def operations(self):
if not we_are_translated(): # For tests
ops = self.short_boxes.values()
@@ -588,7 +660,7 @@
if not isinstance(oldbox, Const) and newbox not in self.short_boxes:
self.short_boxes[newbox] = self.short_boxes[oldbox]
self.aliases[newbox] = oldbox
-
+
def original(self, box):
while box in self.aliases:
box = self.aliases[box]
diff --git a/pypy/jit/metainterp/optimizeopt/vstring.py b/pypy/jit/metainterp/optimizeopt/vstring.py
--- a/pypy/jit/metainterp/optimizeopt/vstring.py
+++ b/pypy/jit/metainterp/optimizeopt/vstring.py
@@ -163,17 +163,6 @@
for value in self._chars:
value.get_args_for_fail(modifier)
- def FIXME_enum_forced_boxes(self, boxes, already_seen):
- key = self.get_key_box()
- if key in already_seen:
- return
- already_seen[key] = None
- if self.box is None:
- for box in self._chars:
- box.enum_forced_boxes(boxes, already_seen)
- else:
- boxes.append(self.box)
-
def _make_virtual(self, modifier):
return modifier.make_vstrplain(self.mode is mode_unicode)
@@ -226,18 +215,6 @@
self.left.get_args_for_fail(modifier)
self.right.get_args_for_fail(modifier)
- def FIXME_enum_forced_boxes(self, boxes, already_seen):
- key = self.get_key_box()
- if key in already_seen:
- return
- already_seen[key] = None
- if self.box is None:
- self.left.enum_forced_boxes(boxes, already_seen)
- self.right.enum_forced_boxes(boxes, already_seen)
- self.lengthbox = None
- else:
- boxes.append(self.box)
-
def _make_virtual(self, modifier):
return modifier.make_vstrconcat(self.mode is mode_unicode)
@@ -284,18 +261,6 @@
self.vstart.get_args_for_fail(modifier)
self.vlength.get_args_for_fail(modifier)
- def FIXME_enum_forced_boxes(self, boxes, already_seen):
- key = self.get_key_box()
- if key in already_seen:
- return
- already_seen[key] = None
- if self.box is None:
- self.vstr.enum_forced_boxes(boxes, already_seen)
- self.vstart.enum_forced_boxes(boxes, already_seen)
- self.vlength.enum_forced_boxes(boxes, already_seen)
- else:
- boxes.append(self.box)
-
def _make_virtual(self, modifier):
return modifier.make_vstrslice(self.mode is mode_unicode)
diff --git a/pypy/jit/metainterp/pyjitpl.py b/pypy/jit/metainterp/pyjitpl.py
--- a/pypy/jit/metainterp/pyjitpl.py
+++ b/pypy/jit/metainterp/pyjitpl.py
@@ -240,8 +240,8 @@
return self.execute(rop.PTR_EQ, box, history.CONST_NULL)
@arguments("box")
- def opimpl_cast_opaque_ptr(self, box):
- return self.execute(rop.CAST_OPAQUE_PTR, box)
+ def opimpl_mark_opaque_ptr(self, box):
+ return self.execute(rop.MARK_OPAQUE_PTR, box)
@arguments("box")
def _opimpl_any_return(self, box):
diff --git a/pypy/jit/metainterp/resoperation.py b/pypy/jit/metainterp/resoperation.py
--- a/pypy/jit/metainterp/resoperation.py
+++ b/pypy/jit/metainterp/resoperation.py
@@ -439,7 +439,6 @@
'PTR_NE/2b',
'INSTANCE_PTR_EQ/2b',
'INSTANCE_PTR_NE/2b',
- 'CAST_OPAQUE_PTR/1b',
#
'ARRAYLEN_GC/1d',
'STRLEN/1',
@@ -471,6 +470,7 @@
'FORCE_TOKEN/0',
'VIRTUAL_REF/2', # removed before it's passed to the backend
'READ_TIMESTAMP/0',
+ 'MARK_OPAQUE_PTR/1b',
'_NOSIDEEFFECT_LAST', # ----- end of no_side_effect operations -----
'SETARRAYITEM_GC/3d',
diff --git a/pypy/jit/metainterp/resume.py b/pypy/jit/metainterp/resume.py
--- a/pypy/jit/metainterp/resume.py
+++ b/pypy/jit/metainterp/resume.py
@@ -139,7 +139,7 @@
self.numberings = {}
self.cached_boxes = {}
self.cached_virtuals = {}
-
+
self.nvirtuals = 0
self.nvholes = 0
self.nvreused = 0
@@ -273,6 +273,9 @@
def make_varray(self, arraydescr):
return VArrayInfo(arraydescr)
+ def make_varraystruct(self, arraydescr, fielddescrs):
+ return VArrayStructInfo(arraydescr, fielddescrs)
+
def make_vstrplain(self, is_unicode=False):
if is_unicode:
return VUniPlainInfo()
@@ -402,7 +405,7 @@
virtuals[num] = vinfo
if self._invalidation_needed(len(liveboxes), nholes):
- memo.clear_box_virtual_numbers()
+ memo.clear_box_virtual_numbers()
def _invalidation_needed(self, nliveboxes, nholes):
memo = self.memo
@@ -455,7 +458,7 @@
def debug_prints(self):
raise NotImplementedError
-
+
class AbstractVirtualStructInfo(AbstractVirtualInfo):
def __init__(self, fielddescrs):
self.fielddescrs = fielddescrs
@@ -537,6 +540,29 @@
for i in self.fieldnums:
debug_print("\t\t", str(untag(i)))
+
+class VArrayStructInfo(AbstractVirtualInfo):
+ def __init__(self, arraydescr, fielddescrs):
+ self.arraydescr = arraydescr
+ self.fielddescrs = fielddescrs
+
+ def debug_prints(self):
+ debug_print("\tvarraystructinfo", self.arraydescr)
+ for i in self.fieldnums:
+ debug_print("\t\t", str(untag(i)))
+
+ @specialize.argtype(1)
+ def allocate(self, decoder, index):
+ array = decoder.allocate_array(self.arraydescr, len(self.fielddescrs))
+ decoder.virtuals_cache[index] = array
+ p = 0
+ for i in range(len(self.fielddescrs)):
+ for j in range(len(self.fielddescrs[i])):
+ decoder.setinteriorfield(i, self.fielddescrs[i][j], array, self.fieldnums[p])
+ p += 1
+ return array
+
+
class VStrPlainInfo(AbstractVirtualInfo):
"""Stands for the string made out of the characters of all fieldnums."""
@@ -884,6 +910,17 @@
self.metainterp.execute_and_record(rop.SETFIELD_GC, descr,
structbox, fieldbox)
+ def setinteriorfield(self, index, descr, array, fieldnum):
+ if descr.is_pointer_field():
+ kind = REF
+ elif descr.is_float_field():
+ kind = FLOAT
+ else:
+ kind = INT
+ fieldbox = self.decode_box(fieldnum, kind)
+ self.metainterp.execute_and_record(rop.SETINTERIORFIELD_GC, descr,
+ array, ConstInt(index), fieldbox)
+
def setarrayitem_int(self, arraydescr, arraybox, index, fieldnum):
self._setarrayitem(arraydescr, arraybox, index, fieldnum, INT)
@@ -1164,6 +1201,17 @@
newvalue = self.decode_int(fieldnum)
self.cpu.bh_setfield_gc_i(struct, descr, newvalue)
+ def setinteriorfield(self, index, descr, array, fieldnum):
+ if descr.is_pointer_field():
+ newvalue = self.decode_ref(fieldnum)
+ self.cpu.bh_setinteriorfield_gc_r(array, index, descr, newvalue)
+ elif descr.is_float_field():
+ newvalue = self.decode_float(fieldnum)
+ self.cpu.bh_setinteriorfield_gc_f(array, index, descr, newvalue)
+ else:
+ newvalue = self.decode_int(fieldnum)
+ self.cpu.bh_setinteriorfield_gc_i(array, index, descr, newvalue)
+
def setarrayitem_int(self, arraydescr, array, index, fieldnum):
newvalue = self.decode_int(fieldnum)
self.cpu.bh_setarrayitem_gc_i(arraydescr, array, index, newvalue)
diff --git a/pypy/jit/metainterp/test/test_ajit.py b/pypy/jit/metainterp/test/test_ajit.py
--- a/pypy/jit/metainterp/test/test_ajit.py
+++ b/pypy/jit/metainterp/test/test_ajit.py
@@ -10,6 +10,7 @@
from pypy.jit.metainterp.typesystem import LLTypeHelper, OOTypeHelper
from pypy.jit.metainterp.warmspot import get_stats
from pypy.jit.metainterp.warmstate import set_future_value
+from pypy.rlib import rerased
from pypy.rlib.jit import (JitDriver, we_are_jitted, hint, dont_look_inside,
loop_invariant, elidable, promote, jit_debug, assert_green,
AssertGreenFailed, unroll_safe, current_trace_length, look_inside_iff,
@@ -3494,16 +3495,70 @@
d = None
while n > 0:
myjitdriver.jit_merge_point(n=n, d=d)
- d = {}
+ d = {"q": 1}
if n % 2:
d["k"] = n
else:
d["z"] = n
- n -= len(d)
+ n -= len(d) - d["q"]
return n
res = self.meta_interp(f, [10])
assert res == 0
+ def test_virtual_dict_constant_keys(self):
+ myjitdriver = JitDriver(greens = [], reds = ["n"])
+ def g(d):
+ return d["key"] - 1
+
+ def f(n):
+ while n > 0:
+ myjitdriver.jit_merge_point(n=n)
+ n = g({"key": n})
+ return n
+
+ res = self.meta_interp(f, [10])
+ assert res == 0
+ self.check_loops({"int_sub": 1, "int_gt": 1, "guard_true": 1, "jump": 1})
+
+ def test_virtual_opaque_ptr(self):
+ myjitdriver = JitDriver(greens = [], reds = ["n"])
+ erase, unerase = rerased.new_erasing_pair("x")
+ @look_inside_iff(lambda x: isvirtual(x))
+ def g(x):
+ return x[0]
+ def f(n):
+ while n > 0:
+ myjitdriver.jit_merge_point(n=n)
+ x = []
+ y = erase(x)
+ z = unerase(y)
+ z.append(1)
+ n -= g(z)
+ return n
+ res = self.meta_interp(f, [10])
+ assert res == 0
+ self.check_loops({"int_sub": 1, "int_gt": 1, "guard_true": 1, "jump": 1})
+
+ def test_virtual_opaque_dict(self):
+ myjitdriver = JitDriver(greens = [], reds = ["n"])
+ erase, unerase = rerased.new_erasing_pair("x")
+ @look_inside_iff(lambda x: isvirtual(x))
+ def g(x):
+ return x[0]["key"] - 1
+ def f(n):
+ while n > 0:
+ myjitdriver.jit_merge_point(n=n)
+ x = [{}]
+ x[0]["key"] = n
+ x[0]["other key"] = n
+ y = erase(x)
+ z = unerase(y)
+ n = g(x)
+ return n
+ res = self.meta_interp(f, [10])
+ assert res == 0
+ self.check_loops({"int_sub": 1, "int_gt": 1, "guard_true": 1, "jump": 1})
+
class TestLLtype(BaseLLtypeTests, LLJitMixin):
@@ -3561,8 +3616,7 @@
res = self.meta_interp(main, [False, 100, True], taggedpointers=True)
def test_rerased(self):
- from pypy.rlib.rerased import erase_int, unerase_int, new_erasing_pair
- eraseX, uneraseX = new_erasing_pair("X")
+ eraseX, uneraseX = rerased.new_erasing_pair("X")
#
class X:
def __init__(self, a, b):
@@ -3575,14 +3629,14 @@
e = eraseX(X(i, j))
else:
try:
- e = erase_int(i)
+ e = rerased.erase_int(i)
except OverflowError:
return -42
if j & 1:
x = uneraseX(e)
return x.a - x.b
else:
- return unerase_int(e)
+ return rerased.unerase_int(e)
#
x = self.interp_operations(f, [-128, 0], taggedpointers=True)
assert x == -128
diff --git a/pypy/jit/metainterp/test/test_heapcache.py b/pypy/jit/metainterp/test/test_heapcache.py
--- a/pypy/jit/metainterp/test/test_heapcache.py
+++ b/pypy/jit/metainterp/test/test_heapcache.py
@@ -371,3 +371,17 @@
assert h.is_unescaped(box1)
h.invalidate_caches(rop.SETARRAYITEM_GC, None, [box2, index1, box1])
assert not h.is_unescaped(box1)
+
+ h = HeapCache()
+ h.new_array(box1, lengthbox1)
+ h.new(box2)
+ assert h.is_unescaped(box1)
+ assert h.is_unescaped(box2)
+ h.invalidate_caches(rop.SETARRAYITEM_GC, None, [box1, lengthbox2, box2])
+ assert h.is_unescaped(box1)
+ assert h.is_unescaped(box2)
+ h.invalidate_caches(
+ rop.CALL, FakeCallDescr(FakeEffektinfo.EF_RANDOM_EFFECTS), [box1]
+ )
+ assert not h.is_unescaped(box1)
+ assert not h.is_unescaped(box2)
diff --git a/pypy/jit/metainterp/test/test_tracingopts.py b/pypy/jit/metainterp/test/test_tracingopts.py
--- a/pypy/jit/metainterp/test/test_tracingopts.py
+++ b/pypy/jit/metainterp/test/test_tracingopts.py
@@ -3,6 +3,7 @@
from pypy.jit.metainterp.test.support import LLJitMixin
from pypy.rlib import jit
from pypy.rlib.rarithmetic import ovfcheck
+from pypy.rlib.rstring import StringBuilder
import py
@@ -590,4 +591,14 @@
assert res == 4
self.check_operations_history(int_add_ovf=0)
res = self.interp_operations(fn, [sys.maxint])
- assert res == 12
\ No newline at end of file
+ assert res == 12
+
+ def test_copy_str_content(self):
+ def fn(n):
+ a = StringBuilder()
+ x = [1]
+ a.append("hello world")
+ return x[0]
+ res = self.interp_operations(fn, [0])
+ assert res == 1
+ self.check_operations_history(getarrayitem_gc=0, getarrayitem_gc_pure=0 )
\ No newline at end of file
diff --git a/pypy/jit/metainterp/warmspot.py b/pypy/jit/metainterp/warmspot.py
--- a/pypy/jit/metainterp/warmspot.py
+++ b/pypy/jit/metainterp/warmspot.py
@@ -62,7 +62,7 @@
clear_tcache()
return jittify_and_run(interp, graph, args, backendopt=backendopt, **kwds)
-def jittify_and_run(interp, graph, args, repeat=1,
+def jittify_and_run(interp, graph, args, repeat=1, graph_and_interp_only=False,
backendopt=False, trace_limit=sys.maxint,
inline=False, loop_longevity=0, retrace_limit=5,
function_threshold=4,
@@ -93,6 +93,8 @@
jd.warmstate.set_param_max_retrace_guards(max_retrace_guards)
jd.warmstate.set_param_enable_opts(enable_opts)
warmrunnerdesc.finish()
+ if graph_and_interp_only:
+ return interp, graph
res = interp.eval_graph(graph, args)
if not kwds.get('translate_support_code', False):
warmrunnerdesc.metainterp_sd.profiler.finish()
@@ -157,6 +159,9 @@
def get_stats():
return pyjitpl._warmrunnerdesc.stats
+def reset_stats():
+ pyjitpl._warmrunnerdesc.stats.clear()
+
def get_translator():
return pyjitpl._warmrunnerdesc.translator
diff --git a/pypy/module/micronumpy/compile.py b/pypy/module/micronumpy/compile.py
--- a/pypy/module/micronumpy/compile.py
+++ b/pypy/module/micronumpy/compile.py
@@ -4,30 +4,55 @@
"""
from pypy.interpreter.baseobjspace import InternalSpaceCache, W_Root
-from pypy.module.micronumpy.interp_dtype import W_Float64Dtype
-from pypy.module.micronumpy.interp_numarray import Scalar, NDimArray, BaseArray
+from pypy.module.micronumpy.interp_dtype import W_Float64Dtype, W_BoolDtype
+from pypy.module.micronumpy.interp_numarray import (Scalar, BaseArray,
+ descr_new_array, scalar_w, NDimArray)
+from pypy.module.micronumpy import interp_ufuncs
from pypy.rlib.objectmodel import specialize
class BogusBytecode(Exception):
pass
-def create_array(dtype, size):
- a = NDimArray(size, dtype=dtype)
- for i in range(size):
- dtype.setitem(a.storage, i, dtype.box(float(i % 10)))
- return a
+class ArgumentMismatch(Exception):
+ pass
+
+class ArgumentNotAnArray(Exception):
+ pass
+
+class WrongFunctionName(Exception):
+ pass
+
+SINGLE_ARG_FUNCTIONS = ["sum", "prod", "max", "min", "all", "any", "unegative"]
class FakeSpace(object):
w_ValueError = None
w_TypeError = None
+ w_None = None
+
+ w_bool = "bool"
+ w_int = "int"
+ w_float = "float"
+ w_list = "list"
+ w_long = "long"
+ w_tuple = 'tuple'
+ w_slice = "slice"
def __init__(self):
"""NOT_RPYTHON"""
self.fromcache = InternalSpaceCache(self).getorbuild
+ self.w_float64dtype = W_Float64Dtype(self)
def issequence_w(self, w_obj):
- return True
+ return isinstance(w_obj, ListObject) or isinstance(w_obj, NDimArray)
+
+ def isinstance_w(self, w_obj, w_tp):
+ if w_obj.tp == w_tp:
+ return True
+ return False
+
+ def decode_index4(self, w_idx, size):
+ return (self.int_w(w_idx), 0, 0, 1)
@specialize.argtype(1)
def wrap(self, obj):
@@ -39,72 +64,391 @@
return IntObject(obj)
raise Exception
+ def newlist(self, items):
+ return ListObject(items)
+
+ def listview(self, obj):
+ assert isinstance(obj, ListObject)
+ return obj.items
+
def float(self, w_obj):
assert isinstance(w_obj, FloatObject)
return w_obj
def float_w(self, w_obj):
+ assert isinstance(w_obj, FloatObject)
return w_obj.floatval
+ def int_w(self, w_obj):
+ if isinstance(w_obj, IntObject):
+ return w_obj.intval
+ elif isinstance(w_obj, FloatObject):
+ return int(w_obj.floatval)
+ raise NotImplementedError
+
+ def int(self, w_obj):
+ return w_obj
+
+ def is_true(self, w_obj):
+ assert isinstance(w_obj, BoolObject)
+ return w_obj.boolval
+
+ def is_w(self, w_obj, w_what):
+ return w_obj is w_what
+
+ def type(self, w_obj):
+ return w_obj.tp
+
+ def gettypefor(self, w_obj):
+ return None
+
+ def call_function(self, tp, w_dtype):
+ return w_dtype
+
+ @specialize.arg(1)
+ def interp_w(self, tp, what):
+ assert isinstance(what, tp)
+ return what
+
+ def len_w(self, w_obj):
+ if isinstance(w_obj, ListObject):
+ return len(w_obj.items)
+ # XXX array probably
+ assert False
class FloatObject(W_Root):
+ tp = FakeSpace.w_float
def __init__(self, floatval):
self.floatval = floatval
class BoolObject(W_Root):
+ tp = FakeSpace.w_bool
def __init__(self, boolval):
self.boolval = boolval
class IntObject(W_Root):
+ tp = FakeSpace.w_int
def __init__(self, intval):
self.intval = intval
+class ListObject(W_Root):
+ tp = FakeSpace.w_list
+ def __init__(self, items):
+ self.items = items
-space = FakeSpace()
+class InterpreterState(object):
+ def __init__(self, code):
+ self.code = code
+ self.variables = {}
+ self.results = []
-def numpy_compile(bytecode, array_size):
- stack = []
- i = 0
- dtype = space.fromcache(W_Float64Dtype)
- for b in bytecode:
- if b == 'a':
- stack.append(create_array(dtype, array_size))
- i += 1
- elif b == 'f':
- stack.append(Scalar(dtype, dtype.box(1.2)))
- elif b == '+':
- right = stack.pop()
- res = stack.pop().descr_add(space, right)
- assert isinstance(res, BaseArray)
- stack.append(res)
- elif b == '-':
- right = stack.pop()
- res = stack.pop().descr_sub(space, right)
- assert isinstance(res, BaseArray)
- stack.append(res)
- elif b == '*':
- right = stack.pop()
- res = stack.pop().descr_mul(space, right)
- assert isinstance(res, BaseArray)
- stack.append(res)
- elif b == '/':
- right = stack.pop()
- res = stack.pop().descr_div(space, right)
- assert isinstance(res, BaseArray)
- stack.append(res)
- elif b == '%':
- right = stack.pop()
- res = stack.pop().descr_mod(space, right)
- assert isinstance(res, BaseArray)
- stack.append(res)
- elif b == '|':
- res = stack.pop().descr_abs(space)
- assert isinstance(res, BaseArray)
- stack.append(res)
+ def run(self, space):
+ self.space = space
+ for stmt in self.code.statements:
+ stmt.execute(self)
+
+class Node(object):
+ def __eq__(self, other):
+ return (self.__class__ == other.__class__ and
+ self.__dict__ == other.__dict__)
+
+ def __ne__(self, other):
+ return not self == other
+
+ def wrap(self, space):
+ raise NotImplementedError
+
+ def execute(self, interp):
+ raise NotImplementedError
+
+class Assignment(Node):
+ def __init__(self, name, expr):
+ self.name = name
+ self.expr = expr
+
+ def execute(self, interp):
+ interp.variables[self.name] = self.expr.execute(interp)
+
+ def __repr__(self):
+ return "%% = %r" % (self.name, self.expr)
+
+class ArrayAssignment(Node):
+ def __init__(self, name, index, expr):
+ self.name = name
+ self.index = index
+ self.expr = expr
+
+ def execute(self, interp):
+ arr = interp.variables[self.name]
+ w_index = self.index.execute(interp).eval(0).wrap(interp.space)
+ # cast to int
+ if isinstance(w_index, FloatObject):
+ w_index = IntObject(int(w_index.floatval))
+ w_val = self.expr.execute(interp).eval(0).wrap(interp.space)
+ arr.descr_setitem(interp.space, w_index, w_val)
+
+ def __repr__(self):
+ return "%s[%r] = %r" % (self.name, self.index, self.expr)
+
+class Variable(Node):
+ def __init__(self, name):
+ self.name = name
+
+ def execute(self, interp):
+ return interp.variables[self.name]
+
+ def __repr__(self):
+ return 'v(%s)' % self.name
+
+class Operator(Node):
+ def __init__(self, lhs, name, rhs):
+ self.name = name
+ self.lhs = lhs
+ self.rhs = rhs
+
+ def execute(self, interp):
+ w_lhs = self.lhs.execute(interp)
+ assert isinstance(w_lhs, BaseArray)
+ if isinstance(self.rhs, SliceConstant):
+ # XXX interface has changed on multidim branch
+ raise NotImplementedError
+ w_rhs = self.rhs.execute(interp)
+ if self.name == '+':
+ w_res = w_lhs.descr_add(interp.space, w_rhs)
+ elif self.name == '*':
+ w_res = w_lhs.descr_mul(interp.space, w_rhs)
+ elif self.name == '-':
+ w_res = w_lhs.descr_sub(interp.space, w_rhs)
+ elif self.name == '->':
+ if isinstance(w_rhs, Scalar):
+ index = int(interp.space.float_w(
+ w_rhs.value.wrap(interp.space)))
+ dtype = interp.space.fromcache(W_Float64Dtype)
+ return Scalar(dtype, w_lhs.get_concrete().eval(index))
+ else:
+ raise NotImplementedError
else:
- print "Unknown opcode: %s" % b
- raise BogusBytecode()
- if len(stack) != 1:
- print "Bogus bytecode, uneven stack length"
- raise BogusBytecode()
- return stack[0]
+ raise NotImplementedError
+ if not isinstance(w_res, BaseArray):
+ dtype = interp.space.fromcache(W_Float64Dtype)
+ w_res = scalar_w(interp.space, dtype, w_res)
+ return w_res
+
+ def __repr__(self):
+ return '(%r %s %r)' % (self.lhs, self.name, self.rhs)
+
+class FloatConstant(Node):
+ def __init__(self, v):
+ self.v = float(v)
+
+ def __repr__(self):
+ return "Const(%s)" % self.v
+
+ def wrap(self, space):
+ return space.wrap(self.v)
+
+ def execute(self, interp):
+ dtype = interp.space.fromcache(W_Float64Dtype)
+ assert isinstance(dtype, W_Float64Dtype)
+ return Scalar(dtype, dtype.box(self.v))
+
+class RangeConstant(Node):
+ def __init__(self, v):
+ self.v = int(v)
+
+ def execute(self, interp):
+ w_list = interp.space.newlist(
+ [interp.space.wrap(float(i)) for i in range(self.v)])
+ dtype = interp.space.fromcache(W_Float64Dtype)
+ return descr_new_array(interp.space, None, w_list, w_dtype=dtype)
+
+ def __repr__(self):
+ return 'Range(%s)' % self.v
+
+class Code(Node):
+ def __init__(self, statements):
+ self.statements = statements
+
+ def __repr__(self):
+ return "\n".join([repr(i) for i in self.statements])
+
+class ArrayConstant(Node):
+ def __init__(self, items):
+ self.items = items
+
+ def wrap(self, space):
+ return space.newlist([item.wrap(space) for item in self.items])
+
+ def execute(self, interp):
+ w_list = self.wrap(interp.space)
+ dtype = interp.space.fromcache(W_Float64Dtype)
+ return descr_new_array(interp.space, None, w_list, w_dtype=dtype)
+
+ def __repr__(self):
+ return "[" + ", ".join([repr(item) for item in self.items]) + "]"
+
+class SliceConstant(Node):
+ def __init__(self):
+ pass
+
+ def __repr__(self):
+ return 'slice()'
+
+class Execute(Node):
+ def __init__(self, expr):
+ self.expr = expr
+
+ def __repr__(self):
+ return repr(self.expr)
+
+ def execute(self, interp):
+ interp.results.append(self.expr.execute(interp))
+
+class FunctionCall(Node):
+ def __init__(self, name, args):
+ self.name = name
+ self.args = args
+
+ def __repr__(self):
+ return "%s(%s)" % (self.name, ", ".join([repr(arg)
+ for arg in self.args]))
+
+ def execute(self, interp):
+ if self.name in SINGLE_ARG_FUNCTIONS:
+ if len(self.args) != 1:
+ raise ArgumentMismatch
+ arr = self.args[0].execute(interp)
+ if not isinstance(arr, BaseArray):
+ raise ArgumentNotAnArray
+ if self.name == "sum":
+ w_res = arr.descr_sum(interp.space)
+ elif self.name == "prod":
+ w_res = arr.descr_prod(interp.space)
+ elif self.name == "max":
+ w_res = arr.descr_max(interp.space)
+ elif self.name == "min":
+ w_res = arr.descr_min(interp.space)
+ elif self.name == "any":
+ w_res = arr.descr_any(interp.space)
+ elif self.name == "all":
+ w_res = arr.descr_all(interp.space)
+ elif self.name == "unegative":
+ neg = interp_ufuncs.get(interp.space).negative
+ w_res = neg.call(interp.space, [arr])
+ else:
+ assert False # unreachable code
+ if isinstance(w_res, BaseArray):
+ return w_res
+ if isinstance(w_res, FloatObject):
+ dtype = interp.space.fromcache(W_Float64Dtype)
+ elif isinstance(w_res, BoolObject):
+ dtype = interp.space.fromcache(W_BoolDtype)
+ else:
+ dtype = None
+ return scalar_w(interp.space, dtype, w_res)
+ else:
+ raise WrongFunctionName
+
+class Parser(object):
+ def parse_identifier(self, id):
+ id = id.strip(" ")
+ #assert id.isalpha()
+ return Variable(id)
+
+ def parse_expression(self, expr):
+ tokens = [i for i in expr.split(" ") if i]
+ if len(tokens) == 1:
+ return self.parse_constant_or_identifier(tokens[0])
+ stack = []
+ tokens.reverse()
+ while tokens:
+ token = tokens.pop()
+ if token == ')':
+ raise NotImplementedError
+ elif self.is_identifier_or_const(token):
+ if stack:
+ name = stack.pop().name
+ lhs = stack.pop()
+ rhs = self.parse_constant_or_identifier(token)
+ stack.append(Operator(lhs, name, rhs))
+ else:
+ stack.append(self.parse_constant_or_identifier(token))
+ else:
+ stack.append(Variable(token))
+ assert len(stack) == 1
+ return stack[-1]
+
+ def parse_constant(self, v):
+ lgt = len(v)-1
+ assert lgt >= 0
+ if ':' in v:
+ # a slice
+ assert v == ':'
+ return SliceConstant()
+ if v[0] == '[':
+ return ArrayConstant([self.parse_constant(elem)
+ for elem in v[1:lgt].split(",")])
+ if v[0] == '|':
+ return RangeConstant(v[1:lgt])
+ return FloatConstant(v)
+
+ def is_identifier_or_const(self, v):
+ c = v[0]
+ if ((c >= 'a' and c <= 'z') or (c >= 'A' and c <= 'Z') or
+ (c >= '0' and c <= '9') or c in '-.[|:'):
+ if v == '-' or v == "->":
+ return False
+ return True
+ return False
+
+ def parse_function_call(self, v):
+ l = v.split('(')
+ assert len(l) == 2
+ name = l[0]
+ cut = len(l[1]) - 1
+ assert cut >= 0
+ args = [self.parse_constant_or_identifier(id)
+ for id in l[1][:cut].split(",")]
+ return FunctionCall(name, args)
+
+ def parse_constant_or_identifier(self, v):
+ c = v[0]
+ if (c >= 'a' and c <= 'z') or (c >= 'A' and c <= 'Z'):
+ if '(' in v:
+ return self.parse_function_call(v)
+ return self.parse_identifier(v)
+ return self.parse_constant(v)
+
+ def parse_array_subscript(self, v):
+ v = v.strip(" ")
+ l = v.split("[")
+ lgt = len(l[1]) - 1
+ assert lgt >= 0
+ rhs = self.parse_constant_or_identifier(l[1][:lgt])
+ return l[0], rhs
+
+ def parse_statement(self, line):
+ if '=' in line:
+ lhs, rhs = line.split("=")
+ lhs = lhs.strip(" ")
+ if '[' in lhs:
+ name, index = self.parse_array_subscript(lhs)
+ return ArrayAssignment(name, index, self.parse_expression(rhs))
+ else:
+ return Assignment(lhs, self.parse_expression(rhs))
+ else:
+ return Execute(self.parse_expression(line))
+
+ def parse(self, code):
+ statements = []
+ for line in code.split("\n"):
+ if '#' in line:
+ line = line.split('#', 1)[0]
+ line = line.strip(" ")
+ if line:
+ statements.append(self.parse_statement(line))
+ return Code(statements)
+
+def numpy_compile(code):
+ parser = Parser()
+ return InterpreterState(parser.parse(code))
diff --git a/pypy/module/micronumpy/interp_dtype.py b/pypy/module/micronumpy/interp_dtype.py
--- a/pypy/module/micronumpy/interp_dtype.py
+++ b/pypy/module/micronumpy/interp_dtype.py
@@ -108,6 +108,12 @@
def setitem_w(self, space, storage, i, w_item):
self.setitem(storage, i, self.unwrap(space, w_item))
+ def fill(self, storage, item, start, stop):
+ storage = self.unerase(storage)
+ item = self.unbox(item)
+ for i in xrange(start, stop):
+ storage[i] = item
+
@specialize.argtype(1)
def adapt_val(self, val):
return self.box(rffi.cast(TP.TO.OF, val))
diff --git a/pypy/module/micronumpy/interp_numarray.py b/pypy/module/micronumpy/interp_numarray.py
--- a/pypy/module/micronumpy/interp_numarray.py
+++ b/pypy/module/micronumpy/interp_numarray.py
@@ -37,6 +37,8 @@
batch = space.listview(w_iterable)
while True:
new_batch = []
+ if not batch:
+ return shape, []
if not space.issequence_w(batch[0]):
for elem in batch:
if space.issequence_w(elem):
@@ -52,6 +54,23 @@
shape.append(size)
batch = new_batch
+def descr_new_array(space, w_subtype, w_item_or_iterable, w_dtype=None):
+ # find scalar
+ if space.is_w(w_dtype, space.w_None):
+ w_dtype = _find_dtype(space, w_item_or_iterable)
+ dtype = space.interp_w(interp_dtype.W_Dtype,
+ space.call_function(space.gettypefor(interp_dtype.W_Dtype), w_dtype)
+ )
+ if not space.issequence_w(w_item_or_iterable):
+ return scalar_w(space, dtype, w_item_or_iterable)
+ shape, elems_w = _find_shape_and_elems(space, w_item_or_iterable)
+ size = len(elems_w)
+ arr = NDimArray(size, shape, dtype=dtype)
+ i = 0
+ for i, w_elem in enumerate(elems_w):
+ dtype.setitem_w(space, arr.storage, i, w_elem)
+ return arr
+
class BaseArray(Wrappable):
_attrs_ = ["invalidates", "signature"]
@@ -71,23 +90,6 @@
def add_invalidates(self, other):
self.invalidates.append(other)
- def descr__new__(space, w_subtype, w_item_or_iterable, w_dtype=None):
- # find scalar
- if space.is_w(w_dtype, space.w_None):
- w_dtype = _find_dtype(space, w_item_or_iterable)
- dtype = space.interp_w(interp_dtype.W_Dtype,
- space.call_function(space.gettypefor(interp_dtype.W_Dtype), w_dtype)
- )
- if not space.issequence_w(w_item_or_iterable):
- return scalar_w(space, dtype, w_item_or_iterable)
- shape, elems_w = _find_shape_and_elems(space, w_item_or_iterable)
- size = len(elems_w)
- arr = NDimArray(size, shape, dtype=dtype)
- i = 0
- for i, w_elem in enumerate(elems_w):
- dtype.setitem_w(space, arr.storage, i, w_elem)
- return arr
-
def _unaryop_impl(ufunc_name):
def impl(self, space):
return getattr(interp_ufuncs.get(space), ufunc_name).call(space, [self])
@@ -262,6 +264,11 @@
# we assume C ordering for now
if space.isinstance_w(w_idx, space.w_int):
idx = space.int_w(w_idx)
+ if not self.shape:
+ if idx != 0:
+ raise OperationError(space.w_IndexError,
+ space.wrap("index out of range"))
+ return 0
if idx < 0:
idx = self.shape[0] + idx
if idx < 0 or idx >= self.shape[0]:
@@ -294,6 +301,11 @@
is a list of scalars that match the size of shape
"""
shape_len = self.len_of_shape()
+ if shape_len == 0:
+ if not space.isinstance_w(w_idx, space.w_int):
+ raise OperationError(space.w_IndexError, space.wrap(
+ "wrong index"))
+ return True
if shape_len == 1:
if space.isinstance_w(w_idx, space.w_int):
return True
@@ -403,7 +415,7 @@
self.value = value
def find_size(self):
- return 1
+ raise ValueError
def get_concrete(self):
return self
@@ -432,7 +444,7 @@
i = 0
signature = self.signature
result_size = self.find_size()
- result = NDimArray(result_size, [result_size], self.find_dtype())
+ result = NDimArray(result_size, self.shape, self.find_dtype())
while i < result_size:
numpy_driver.jit_merge_point(signature=signature,
result_size=result_size, i=i,
@@ -706,13 +718,12 @@
arr = NDimArray(size, [size], dtype=dtype)
one = dtype.adapt_val(1)
- for i in xrange(size):
- arr.dtype.setitem(arr.storage, i, one)
+ arr.dtype.fill(arr.storage, one, 0, size)
return space.wrap(arr)
BaseArray.typedef = TypeDef(
'numarray',
- __new__ = interp2app(BaseArray.descr__new__.im_func),
+ __new__ = interp2app(descr_new_array),
__len__ = interp2app(BaseArray.descr_len),
diff --git a/pypy/module/micronumpy/interp_ufuncs.py b/pypy/module/micronumpy/interp_ufuncs.py
--- a/pypy/module/micronumpy/interp_ufuncs.py
+++ b/pypy/module/micronumpy/interp_ufuncs.py
@@ -237,22 +237,20 @@
return dt
def find_dtype_for_scalar(space, w_obj, current_guess=None):
- w_type = space.type(w_obj)
-
bool_dtype = space.fromcache(interp_dtype.W_BoolDtype)
long_dtype = space.fromcache(interp_dtype.W_LongDtype)
int64_dtype = space.fromcache(interp_dtype.W_Int64Dtype)
- if space.is_w(w_type, space.w_bool):
+ if space.isinstance_w(w_obj, space.w_bool):
if current_guess is None or current_guess is bool_dtype:
return bool_dtype
return current_guess
- elif space.is_w(w_type, space.w_int):
+ elif space.isinstance_w(w_obj, space.w_int):
if (current_guess is None or current_guess is bool_dtype or
current_guess is long_dtype):
return long_dtype
return current_guess
- elif space.is_w(w_type, space.w_long):
+ elif space.isinstance_w(w_obj, space.w_long):
if (current_guess is None or current_guess is bool_dtype or
current_guess is long_dtype or current_guess is int64_dtype):
return int64_dtype
diff --git a/pypy/module/micronumpy/test/test_compile.py b/pypy/module/micronumpy/test/test_compile.py
new file mode 100644
--- /dev/null
+++ b/pypy/module/micronumpy/test/test_compile.py
@@ -0,0 +1,170 @@
+
+import py
+from pypy.module.micronumpy.compile import *
+
+class TestCompiler(object):
+ def compile(self, code):
+ return numpy_compile(code)
+
+ def test_vars(self):
+ code = """
+ a = 2
+ b = 3
+ """
+ interp = self.compile(code)
+ assert isinstance(interp.code.statements[0], Assignment)
+ assert interp.code.statements[0].name == 'a'
+ assert interp.code.statements[0].expr.v == 2
+ assert interp.code.statements[1].name == 'b'
+ assert interp.code.statements[1].expr.v == 3
+
+ def test_array_literal(self):
+ code = "a = [1,2,3]"
+ interp = self.compile(code)
+ assert isinstance(interp.code.statements[0].expr, ArrayConstant)
+ st = interp.code.statements[0]
+ assert st.expr.items == [FloatConstant(1), FloatConstant(2),
+ FloatConstant(3)]
+
+ def test_array_literal2(self):
+ code = "a = [[1],[2],[3]]"
+ interp = self.compile(code)
+ assert isinstance(interp.code.statements[0].expr, ArrayConstant)
+ st = interp.code.statements[0]
+ assert st.expr.items == [ArrayConstant([FloatConstant(1)]),
+ ArrayConstant([FloatConstant(2)]),
+ ArrayConstant([FloatConstant(3)])]
+
+ def test_expr_1(self):
+ code = "b = a + 1"
+ interp = self.compile(code)
+ assert (interp.code.statements[0].expr ==
+ Operator(Variable("a"), "+", FloatConstant(1)))
+
+ def test_expr_2(self):
+ code = "b = a + b - 3"
+ interp = self.compile(code)
+ assert (interp.code.statements[0].expr ==
+ Operator(Operator(Variable("a"), "+", Variable("b")), "-",
+ FloatConstant(3)))
+
+ def test_expr_3(self):
+ # an equivalent of range
+ code = "a = |20|"
+ interp = self.compile(code)
+ assert interp.code.statements[0].expr == RangeConstant(20)
+
+ def test_expr_only(self):
+ code = "3 + a"
+ interp = self.compile(code)
+ assert interp.code.statements[0] == Execute(
+ Operator(FloatConstant(3), "+", Variable("a")))
+
+ def test_array_access(self):
+ code = "a -> 3"
+ interp = self.compile(code)
+ assert interp.code.statements[0] == Execute(
+ Operator(Variable("a"), "->", FloatConstant(3)))
+
+ def test_function_call(self):
+ code = "sum(a)"
+ interp = self.compile(code)
+ assert interp.code.statements[0] == Execute(
+ FunctionCall("sum", [Variable("a")]))
+
+ def test_comment(self):
+ code = """
+ # some comment
+ a = b + 3 # another comment
+ """
+ interp = self.compile(code)
+ assert interp.code.statements[0] == Assignment(
+ 'a', Operator(Variable('b'), "+", FloatConstant(3)))
+
+class TestRunner(object):
+ def run(self, code):
+ interp = numpy_compile(code)
+ space = FakeSpace()
+ interp.run(space)
+ return interp
+
+ def test_one(self):
+ code = """
+ a = 3
+ b = 4
+ a + b
+ """
+ interp = self.run(code)
+ assert sorted(interp.variables.keys()) == ['a', 'b']
+ assert interp.results[0]
+
+ def test_array_add(self):
+ code = """
+ a = [1,2,3,4]
+ b = [4,5,6,5]
+ a + b
+ """
+ interp = self.run(code)
+ assert interp.results[0]._getnums(False) == ["5.0", "7.0", "9.0", "9.0"]
+
+ def test_array_getitem(self):
+ code = """
+ a = [1,2,3,4]
+ b = [4,5,6,5]
+ a + b -> 3
+ """
+ interp = self.run(code)
+ assert interp.results[0].value.val == 3 + 6
+
+ def test_range_getitem(self):
+ code = """
+ r = |20| + 3
+ r -> 3
+ """
+ interp = self.run(code)
+ assert interp.results[0].value.val == 6
+
+ def test_sum(self):
+ code = """
+ a = [1,2,3,4,5]
+ r = sum(a)
+ r
+ """
+ interp = self.run(code)
+ assert interp.results[0].value.val == 15
+
+ def test_array_write(self):
+ code = """
+ a = [1,2,3,4,5]
+ a[3] = 15
+ a -> 3
+ """
+ interp = self.run(code)
+ assert interp.results[0].value.val == 15
+
+ def test_min(self):
+ interp = self.run("""
+ a = |30|
+ a[15] = -12
+ b = a + a
+ min(b)
+ """)
+ assert interp.results[0].value.val == -24
+
+ def test_max(self):
+ interp = self.run("""
+ a = |30|
+ a[13] = 128
+ b = a + a
+ max(b)
+ """)
+ assert interp.results[0].value.val == 256
+
+ def test_slice(self):
+ py.test.skip("in progress")
+ interp = self.run("""
+ a = [1,2,3,4]
+ b = a -> :
+ b -> 3
+ """)
+ assert interp.results[0].value.val == 3
diff --git a/pypy/module/micronumpy/test/test_ufuncs.py b/pypy/module/micronumpy/test/test_ufuncs.py
--- a/pypy/module/micronumpy/test/test_ufuncs.py
+++ b/pypy/module/micronumpy/test/test_ufuncs.py
@@ -82,6 +82,8 @@
b = negative(a)
a[0] = 5.0
assert b[0] == 5.0
+ a = array(range(30))
+ assert negative(a + a)[3] == -6
def test_abs(self):
from numpy import array, absolute
diff --git a/pypy/module/micronumpy/test/test_zjit.py b/pypy/module/micronumpy/test/test_zjit.py
--- a/pypy/module/micronumpy/test/test_zjit.py
+++ b/pypy/module/micronumpy/test/test_zjit.py
@@ -1,261 +1,206 @@
+
+""" Tests that check if JIT-compiled numpy operations produce reasonably
+good assembler
+"""
+
+
+import py
from pypy.jit.metainterp.test.support import LLJitMixin
from pypy.module.micronumpy import interp_ufuncs, signature
from pypy.module.micronumpy.compile import (numpy_compile, FakeSpace,
- FloatObject, IntObject)
-from pypy.module.micronumpy.interp_dtype import W_Int32Dtype, W_Float64Dtype, W_Int64Dtype, W_UInt64Dtype
-from pypy.module.micronumpy.interp_numarray import (BaseArray, NDimArray,
- NDimSlice, scalar_w)
+ FloatObject, IntObject, BoolObject)
+from pypy.module.micronumpy.interp_numarray import NDimArray, NDimSlice
from pypy.rlib.nonconst import NonConstant
-from pypy.rpython.annlowlevel import llstr
-from pypy.rpython.test.test_llinterp import interpret
-
-import py
-py.test.skip("XXX")
+from pypy.rpython.annlowlevel import llstr, hlstr
+from pypy.jit.metainterp.warmspot import reset_stats
+from pypy.jit.metainterp import pyjitpl
class TestNumpyJIt(LLJitMixin):
- def setup_class(cls):
- cls.space = FakeSpace()
- cls.float64_dtype = cls.space.fromcache(W_Float64Dtype)
- cls.int64_dtype = cls.space.fromcache(W_Int64Dtype)
- cls.uint64_dtype = cls.space.fromcache(W_UInt64Dtype)
- cls.int32_dtype = cls.space.fromcache(W_Int32Dtype)
+ graph = None
+ interp = None
+
+ def run(self, code):
+ space = FakeSpace()
+
+ def f(code):
+ interp = numpy_compile(hlstr(code))
+ interp.run(space)
+ res = interp.results[-1]
+ w_res = res.eval(0).wrap(interp.space)
+ if isinstance(w_res, BoolObject):
+ return float(w_res.boolval)
+ elif isinstance(w_res, FloatObject):
+ return w_res.floatval
+ elif isinstance(w_res, IntObject):
+ return w_res.intval
+ else:
+ return -42.
+
+ if self.graph is None:
+ interp, graph = self.meta_interp(f, [llstr(code)],
+ listops=True,
+ backendopt=True,
+ graph_and_interp_only=True)
+ self.__class__.interp = interp
+ self.__class__.graph = graph
+
+ reset_stats()
+ pyjitpl._warmrunnerdesc.memory_manager.alive_loops.clear()
+ return self.interp.eval_graph(self.graph, [llstr(code)])
def test_add(self):
- def f(i):
- ar = SingleDimArray(i, dtype=self.float64_dtype)
- v = interp_ufuncs.get(self.space).add.call(self.space, [ar, ar])
- return v.get_concrete().eval(3).val
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = |30|
+ b = a + a
+ b -> 3
+ """)
self.check_loops({'getarrayitem_raw': 2, 'float_add': 1,
'setarrayitem_raw': 1, 'int_add': 1,
'int_lt': 1, 'guard_true': 1, 'jump': 1})
- assert result == f(5)
+ assert result == 3 + 3
def test_floatadd(self):
- def f(i):
- ar = SingleDimArray(i, dtype=self.float64_dtype)
- v = interp_ufuncs.get(self.space).add.call(self.space, [
- ar,
- scalar_w(self.space, self.float64_dtype, self.space.wrap(4.5))
- ],
- )
- assert isinstance(v, BaseArray)
- return v.get_concrete().eval(3).val
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = |30| + 3
+ a -> 3
+ """)
+ assert result == 3 + 3
self.check_loops({"getarrayitem_raw": 1, "float_add": 1,
"setarrayitem_raw": 1, "int_add": 1,
"int_lt": 1, "guard_true": 1, "jump": 1})
- assert result == f(5)
def test_sum(self):
- space = self.space
- float64_dtype = self.float64_dtype
- int64_dtype = self.int64_dtype
-
- def f(i):
- if NonConstant(False):
- dtype = int64_dtype
- else:
- dtype = float64_dtype
- ar = SingleDimArray(i, dtype=dtype)
- v = ar.descr_add(space, ar).descr_sum(space)
- assert isinstance(v, FloatObject)
- return v.floatval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = |30|
+ b = a + a
+ sum(b)
+ """)
+ assert result == 2 * sum(range(30))
self.check_loops({"getarrayitem_raw": 2, "float_add": 2,
"int_add": 1,
"int_lt": 1, "guard_true": 1, "jump": 1})
- assert result == f(5)
def test_prod(self):
- space = self.space
- float64_dtype = self.float64_dtype
- int64_dtype = self.int64_dtype
-
- def f(i):
- if NonConstant(False):
- dtype = int64_dtype
- else:
- dtype = float64_dtype
- ar = SingleDimArray(i, dtype=dtype)
- v = ar.descr_add(space, ar).descr_prod(space)
- assert isinstance(v, FloatObject)
- return v.floatval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = |30|
+ b = a + a
+ prod(b)
+ """)
+ expected = 1
+ for i in range(30):
+ expected *= i * 2
+ assert result == expected
self.check_loops({"getarrayitem_raw": 2, "float_add": 1,
"float_mul": 1, "int_add": 1,
"int_lt": 1, "guard_true": 1, "jump": 1})
- assert result == f(5)
def test_max(self):
- space = self.space
- float64_dtype = self.float64_dtype
- int64_dtype = self.int64_dtype
-
- def f(i):
- if NonConstant(False):
- dtype = int64_dtype
- else:
- dtype = float64_dtype
- ar = SingleDimArray(i, dtype=dtype)
- j = 0
- while j < i:
- ar.get_concrete().setitem(j, float64_dtype.box(float(j)))
- j += 1
- v = ar.descr_add(space, ar).descr_max(space)
- assert isinstance(v, FloatObject)
- return v.floatval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ py.test.skip("broken, investigate")
+ result = self.run("""
+ a = |30|
+ a[13] = 128
+ b = a + a
+ max(b)
+ """)
+ assert result == 256
self.check_loops({"getarrayitem_raw": 2, "float_add": 1,
- "float_gt": 1, "int_add": 1,
- "int_lt": 1, "guard_true": 1,
- "guard_false": 1, "jump": 1})
- assert result == f(5)
+ "float_mul": 1, "int_add": 1,
+ "int_lt": 1, "guard_true": 1, "jump": 1})
def test_min(self):
- space = self.space
- float64_dtype = self.float64_dtype
- int64_dtype = self.int64_dtype
-
- def f(i):
- if NonConstant(False):
- dtype = int64_dtype
- else:
- dtype = float64_dtype
- ar = SingleDimArray(i, dtype=dtype)
- j = 0
- while j < i:
- ar.get_concrete().setitem(j, float64_dtype.box(float(j)))
- j += 1
- v = ar.descr_add(space, ar).descr_min(space)
- assert isinstance(v, FloatObject)
- return v.floatval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ py.test.skip("broken, investigate")
+ result = self.run("""
+ a = |30|
+ a[15] = -12
+ b = a + a
+ min(b)
+ """)
+ assert result == -24
self.check_loops({"getarrayitem_raw": 2, "float_add": 1,
- "float_lt": 1, "int_add": 1,
- "int_lt": 1, "guard_true": 2,
- "jump": 1})
- assert result == f(5)
-
- def test_argmin(self):
- space = self.space
- float64_dtype = self.float64_dtype
-
- def f(i):
- ar = SingleDimArray(i, dtype=NonConstant(float64_dtype))
- j = 0
- while j < i:
- ar.get_concrete().setitem(j, float64_dtype.box(float(j)))
- j += 1
- return ar.descr_add(space, ar).descr_argmin(space).intval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
- self.check_loops({"getarrayitem_raw": 2, "float_add": 1,
- "float_lt": 1, "int_add": 1,
- "int_lt": 1, "guard_true": 2,
- "jump": 1})
- assert result == f(5)
-
- def test_all(self):
- space = self.space
- float64_dtype = self.float64_dtype
-
- def f(i):
- ar = SingleDimArray(i, dtype=NonConstant(float64_dtype))
- j = 0
- while j < i:
- ar.get_concrete().setitem(j, float64_dtype.box(1.0))
- j += 1
- return ar.descr_add(space, ar).descr_all(space).boolval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
- self.check_loops({"getarrayitem_raw": 2, "float_add": 1,
- "int_add": 1, "float_ne": 1,
- "int_lt": 1, "guard_true": 2, "jump": 1})
- assert result == f(5)
+ "float_mul": 1, "int_add": 1,
+ "int_lt": 1, "guard_true": 1, "jump": 1})
def test_any(self):
- space = self.space
- float64_dtype = self.float64_dtype
-
- def f(i):
- ar = SingleDimArray(i, dtype=NonConstant(float64_dtype))
- return ar.descr_add(space, ar).descr_any(space).boolval
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = [0,0,0,0,0,0,0,0,0,0,0]
+ a[8] = -12
+ b = a + a
+ any(b)
+ """)
+ assert result == 1
self.check_loops({"getarrayitem_raw": 2, "float_add": 1,
- "int_add": 1, "float_ne": 1, "guard_false": 1,
- "int_lt": 1, "guard_true": 1, "jump": 1})
- assert result == f(5)
+ "float_ne": 1, "int_add": 1,
+ "int_lt": 1, "guard_true": 1, "jump": 1,
+ "guard_false": 1})
def test_already_forced(self):
- space = self.space
-
- def f(i):
- ar = SingleDimArray(i, dtype=self.float64_dtype)
- v1 = interp_ufuncs.get(self.space).add.call(space, [ar, scalar_w(space, self.float64_dtype, space.wrap(4.5))])
- assert isinstance(v1, BaseArray)
- v2 = interp_ufuncs.get(self.space).multiply.call(space, [v1, scalar_w(space, self.float64_dtype, space.wrap(4.5))])
- v1.force_if_needed()
- assert isinstance(v2, BaseArray)
- return v2.get_concrete().eval(3).val
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = |30|
+ b = a + 4.5
+ b -> 5 # forces
+ c = b * 8
+ c -> 5
+ """)
+ assert result == (5 + 4.5) * 8
# This is the sum of the ops for both loops, however if you remove the
# optimization then you end up with 2 float_adds, so we can still be
# sure it was optimized correctly.
self.check_loops({"getarrayitem_raw": 2, "float_mul": 1, "float_add": 1,
"setarrayitem_raw": 2, "int_add": 2,
"int_lt": 2, "guard_true": 2, "jump": 2})
- assert result == f(5)
def test_ufunc(self):
- space = self.space
- def f(i):
- ar = SingleDimArray(i, dtype=self.float64_dtype)
- v1 = interp_ufuncs.get(self.space).add.call(space, [ar, ar])
- v2 = interp_ufuncs.get(self.space).negative.call(space, [v1])
- return v2.get_concrete().eval(3).val
-
- result = self.meta_interp(f, [5], listops=True, backendopt=True)
+ result = self.run("""
+ a = |30|
+ b = a + a
+ c = unegative(b)
+ c -> 3
+ """)
+ assert result == -6
self.check_loops({"getarrayitem_raw": 2, "float_add": 1, "float_neg": 1,
"setarrayitem_raw": 1, "int_add": 1,
"int_lt": 1, "guard_true": 1, "jump": 1,
})
- assert result == f(5)
- def test_appropriate_specialization(self):
- space = self.space
- def f(i):
- ar = SingleDimArray(i, dtype=self.float64_dtype)
-
- v1 = interp_ufuncs.get(self.space).add.call(space, [ar, ar])
- v2 = interp_ufuncs.get(self.space).negative.call(space, [v1])
- v2.get_concrete()
-
- for i in xrange(5):
- v1 = interp_ufuncs.get(self.space).multiply.call(space, [ar, ar])
- v2 = interp_ufuncs.get(self.space).negative.call(space, [v1])
- v2.get_concrete()
-
- self.meta_interp(f, [5], listops=True, backendopt=True)
+ def test_specialization(self):
+ self.run("""
+ a = |30|
+ b = a + a
+ c = unegative(b)
+ c -> 3
+ d = a * a
+ unegative(d)
+ d -> 3
+ d = a * a
+ unegative(d)
+ d -> 3
+ d = a * a
+ unegative(d)
+ d -> 3
+ d = a * a
+ unegative(d)
+ d -> 3
+ """)
# This is 3, not 2 because there is a bridge for the exit.
self.check_loop_count(3)
+
+class TestNumpyOld(LLJitMixin):
+ def setup_class(cls):
+ from pypy.module.micronumpy.compile import FakeSpace
+ from pypy.module.micronumpy.interp_dtype import W_Float64Dtype
+
+ cls.space = FakeSpace()
+ cls.float64_dtype = cls.space.fromcache(W_Float64Dtype)
+
def test_slice(self):
def f(i):
step = 3
- ar = SingleDimArray(step*i, dtype=self.float64_dtype)
+ ar = NDimArray(step*i, dtype=self.float64_dtype)
new_sig = signature.Signature.find_sig([
- SingleDimSlice.signature, ar.signature
+ NDimSlice.signature, ar.signature
])
- s = SingleDimSlice(0, step*i, step, i, ar, new_sig)
+ s = NDimSlice(0, step*i, step, i, ar, new_sig)
v = interp_ufuncs.get(self.space).add.call(self.space, [s, s])
return v.get_concrete().eval(3).val
@@ -269,15 +214,15 @@
def f(i):
step1 = 2
step2 = 3
- ar = SingleDimArray(step2*i, dtype=self.float64_dtype)
+ ar = NDimArray(step2*i, dtype=self.float64_dtype)
new_sig = signature.Signature.find_sig([
- SingleDimSlice.signature, ar.signature
+ NDimSlice.signature, ar.signature
])
- s1 = SingleDimSlice(0, step1*i, step1, i, ar, new_sig)
+ s1 = NDimSlice(0, step1*i, step1, i, ar, new_sig)
new_sig = signature.Signature.find_sig([
- SingleDimSlice.signature, s1.signature
+ NDimSlice.signature, s1.signature
])
- s2 = SingleDimSlice(0, step2*i, step2, i, ar, new_sig)
+ s2 = NDimSlice(0, step2*i, step2, i, ar, new_sig)
v = interp_ufuncs.get(self.space).add.call(self.space, [s1, s2])
return v.get_concrete().eval(3).val
@@ -293,8 +238,8 @@
def f(i):
step = NonConstant(3)
- ar = SingleDimArray(step*i, dtype=float64_dtype)
- ar2 = SingleDimArray(i, dtype=float64_dtype)
+ ar = NDimArray(step*i, dtype=float64_dtype)
+ ar2 = NDimArray(i, dtype=float64_dtype)
ar2.get_concrete().setitem(1, float64_dtype.box(5.5))
arg = ar2.descr_add(space, ar2)
ar.setslice(space, 0, step*i, step, i, arg)
@@ -332,17 +277,3 @@
result = self.meta_interp(f, [5], listops=True, backendopt=True)
assert result == f(5)
-class TestTranslation(object):
- def test_compile(self):
- x = numpy_compile('aa+f*f/a-', 10)
- x = x.compute()
- assert isinstance(x, SingleDimArray)
- assert x.size == 10
- assert x.eval(0).val == 0
- assert x.eval(1).val == ((1 + 1) * 1.2) / 1.2 - 1
-
- def test_translation(self):
- # we import main to check if the target compiles
- from pypy.translator.goal.targetnumpystandalone import main
-
- interpret(main, [llstr('af+'), 100])
diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py
--- a/pypy/module/pypyjit/test_pypy_c/test_call.py
+++ b/pypy/module/pypyjit/test_pypy_c/test_call.py
@@ -465,3 +465,25 @@
setfield_gc(p4, p22, descr=<GcPtrFieldDescr pypy.interpreter.nestedscope.Cell.inst_w_value .*>)
jump(p0, p1, p2, p3, p4, p7, p22, p7, descr=<Loop0>)
""")
+
+ def test_kwargs_virtual(self):
+ def main(n):
+ def g(**kwargs):
+ return kwargs["x"] + 1
+
+ i = 0
+ while i < n:
+ i = g(x=i)
+ return i
+
+ log = self.run(main, [500])
+ assert log.result == 500
+ loop, = log.loops_by_filename(self.filepath)
+ assert loop.match("""
+ i2 = int_lt(i0, i1)
+ guard_true(i2, descr=...)
+ i3 = force_token()
+ i4 = int_add(i0, 1)
+ --TICK--
+ jump(..., descr=...)
+ """)
\ No newline at end of file
diff --git a/pypy/module/pypyjit/test_pypy_c/test_containers.py b/pypy/module/pypyjit/test_pypy_c/test_containers.py
--- a/pypy/module/pypyjit/test_pypy_c/test_containers.py
+++ b/pypy/module/pypyjit/test_pypy_c/test_containers.py
@@ -44,7 +44,7 @@
# gc_id call is hoisted out of the loop, the id of a value obviously
# can't change ;)
assert loop.match_by_id("getitem", """
- i28 = call(ConstClass(ll_dict_lookup__dicttablePtr_objectPtr_Signed), p18, p6, i25, descr=...)
+ i26 = call(ConstClass(ll_dict_lookup), p18, p6, i25, descr=...)
...
p33 = getinteriorfield_gc(p31, i26, descr=<InteriorFieldDescr <GcPtrFieldDescr dictentry.value .*>>)
...
diff --git a/pypy/objspace/std/test/test_stdobjspace.py b/pypy/objspace/std/test/test_stdobjspace.py
--- a/pypy/objspace/std/test/test_stdobjspace.py
+++ b/pypy/objspace/std/test/test_stdobjspace.py
@@ -14,11 +14,11 @@
def test_int_w_non_int(self):
raises(OperationError,self.space.int_w,self.space.wrap(None))
- raises(OperationError,self.space.int_w,self.space.wrap(""))
+ raises(OperationError,self.space.int_w,self.space.wrap(""))
def test_uint_w_non_int(self):
raises(OperationError,self.space.uint_w,self.space.wrap(None))
- raises(OperationError,self.space.uint_w,self.space.wrap(""))
+ raises(OperationError,self.space.uint_w,self.space.wrap(""))
def test_multimethods_defined_on(self):
from pypy.objspace.std.stdtypedef import multimethods_defined_on
@@ -49,14 +49,14 @@
def test_fastpath_isinstance(self):
from pypy.objspace.std.stringobject import W_StringObject
from pypy.objspace.std.intobject import W_IntObject
-
+
space = self.space
- assert space.w_str.interplevel_cls is W_StringObject
- assert space.w_int.interplevel_cls is W_IntObject
+ assert space._get_interplevel_cls(space.w_str) is W_StringObject
+ assert space._get_interplevel_cls(space.w_int) is W_IntObject
class X(W_StringObject):
def __init__(self):
pass
-
+
typedef = None
assert space.isinstance_w(X(), space.w_str)
diff --git a/pypy/rlib/rsre/rsre_core.py b/pypy/rlib/rsre/rsre_core.py
--- a/pypy/rlib/rsre/rsre_core.py
+++ b/pypy/rlib/rsre/rsre_core.py
@@ -391,6 +391,8 @@
if self.num_pending >= min:
while enum is not None and ptr == ctx.match_end:
enum = enum.move_to_next_result(ctx)
+ # matched marks for zero-width assertions
+ marks = ctx.match_marks
#
if enum is not None:
# matched one more 'item'. record it and continue.
diff --git a/pypy/rlib/rsre/test/test_re.py b/pypy/rlib/rsre/test/test_re.py
--- a/pypy/rlib/rsre/test/test_re.py
+++ b/pypy/rlib/rsre/test/test_re.py
@@ -226,6 +226,13 @@
(None, 'b', None))
assert pat.match('ac').group(1, 'b2', 3) == ('a', None, 'c')
+ def test_bug_923(self):
+ # Issue923: grouping inside optional lookahead problem
+ assert re.match(r'a(?=(b))?', "ab").groups() == ("b",)
+ assert re.match(r'(a(?=(b))?)', "ab").groups() == ('a', 'b')
+ assert re.match(r'(a)(?=(b))?', "ab").groups() == ('a', 'b')
+ assert re.match(r'(?P<g1>a)(?=(?P<g2>b))?', "ab").groupdict() == {'g1': 'a', 'g2': 'b'}
+
def test_re_groupref_exists(self):
assert re.match('^(\()?([^()]+)(?(1)\))$', '(a)').groups() == (
('(', 'a'))
diff --git a/pypy/rpython/lltypesystem/rdict.py b/pypy/rpython/lltypesystem/rdict.py
--- a/pypy/rpython/lltypesystem/rdict.py
+++ b/pypy/rpython/lltypesystem/rdict.py
@@ -445,9 +445,9 @@
i = ll_dict_lookup(d, key, hash)
return _ll_dict_setitem_lookup_done(d, key, value, hash, i)
-# Leaving as dont_look_inside ATM, it has a few branches which could lead to
-# many bridges if we don't consider their possible frequency.
- at jit.dont_look_inside
+# It may be safe to look inside always, it has a few branches though, and their
+# frequencies needs to be investigated.
+ at jit.look_inside_iff(lambda d, key, value, hash, i: jit.isvirtual(d) and jit.isconstant(key))
def _ll_dict_setitem_lookup_done(d, key, value, hash, i):
valid = (i & HIGHEST_BIT) == 0
i = i & MASK
@@ -533,7 +533,7 @@
# ------- a port of CPython's dictobject.c's lookdict implementation -------
PERTURB_SHIFT = 5
- at jit.dont_look_inside
+ at jit.look_inside_iff(lambda d, key, hash: jit.isvirtual(d) and jit.isconstant(key))
def ll_dict_lookup(d, key, hash):
entries = d.entries
ENTRIES = lltype.typeOf(entries).TO
diff --git a/pypy/rpython/lltypesystem/rstr.py b/pypy/rpython/lltypesystem/rstr.py
--- a/pypy/rpython/lltypesystem/rstr.py
+++ b/pypy/rpython/lltypesystem/rstr.py
@@ -20,6 +20,7 @@
from pypy.rpython.rmodel import Repr
from pypy.rpython.lltypesystem import llmemory
from pypy.tool.sourcetools import func_with_new_name
+from pypy.rpython.lltypesystem.lloperation import llop
# ____________________________________________________________
#
@@ -364,8 +365,10 @@
while lpos < rpos and s.chars[lpos] == ch:
lpos += 1
if right:
- while lpos < rpos and s.chars[rpos] == ch:
+ while lpos < rpos + 1 and s.chars[rpos] == ch:
rpos -= 1
+ if rpos < lpos:
+ return s.empty()
r_len = rpos - lpos + 1
result = s.malloc(r_len)
s.copy_contents(s, result, lpos, 0, r_len)
diff --git a/pypy/rpython/test/test_rstr.py b/pypy/rpython/test/test_rstr.py
--- a/pypy/rpython/test/test_rstr.py
+++ b/pypy/rpython/test/test_rstr.py
@@ -372,12 +372,20 @@
return const('!ab!').lstrip(const('!'))
def right():
return const('!ab!').rstrip(const('!'))
+ def empty():
+ return const(' ').strip(' ')
+ def left2():
+ return const('a ').strip(' ')
res = self.interpret(both, [])
assert self.ll_to_string(res) == const('ab')
res = self.interpret(left, [])
assert self.ll_to_string(res) == const('ab!')
res = self.interpret(right, [])
assert self.ll_to_string(res) == const('!ab')
+ res = self.interpret(empty, [])
+ assert self.ll_to_string(res) == const('')
+ res = self.interpret(left2, [])
+ assert self.ll_to_string(res) == const('a')
def test_upper(self):
const = self.const
More information about the pypy-commit
mailing list