[pypy-svn] r54649 - in pypy/branch/hybrid-io/pypy: . config interpreter objspace/std rlib rlib/test rpython rpython/lltypesystem rpython/lltypesystem/test rpython/memory rpython/memory/gc rpython/memory/gctransform rpython/memory/test rpython/module rpython/module/test rpython/ootypesystem rpython/test translator translator/backendopt translator/backendopt/test translator/c translator/c/test translator/goal translator/llvm

fijal at codespeak.net fijal at codespeak.net
Sun May 11 12:45:50 CEST 2008


Author: fijal
Date: Sun May 11 12:45:47 2008
New Revision: 54649

Added:
   pypy/branch/hybrid-io/pypy/FAILURES
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/FAILURES
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rbuilder.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/rpython/lltypesystem/rbuilder.py
   pypy/branch/hybrid-io/pypy/rpython/ootypesystem/rbuilder.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/rpython/ootypesystem/rbuilder.py
   pypy/branch/hybrid-io/pypy/rpython/rbuilder.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/rpython/rbuilder.py
   pypy/branch/hybrid-io/pypy/rpython/test/test_rbuilder.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/rpython/test/test_rbuilder.py
   pypy/branch/hybrid-io/pypy/translator/goal/targetreadlines.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/translator/goal/targetreadlines.py
   pypy/branch/hybrid-io/pypy/translator/goal/targetsimpleread.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/translator/goal/targetsimpleread.py
   pypy/branch/hybrid-io/pypy/translator/goal/targetsimplewrite.py
      - copied unchanged from r54648, pypy/branch/io-improvements/pypy/translator/goal/targetsimplewrite.py
Removed:
   pypy/branch/hybrid-io/pypy/translator/backendopt/coalloc.py
   pypy/branch/hybrid-io/pypy/translator/backendopt/test/test_coalloc.py
Modified:
   pypy/branch/hybrid-io/pypy/config/translationoption.py
   pypy/branch/hybrid-io/pypy/interpreter/pyframe.py
   pypy/branch/hybrid-io/pypy/interpreter/pyopcode.py
   pypy/branch/hybrid-io/pypy/objspace/std/formatting.py
   pypy/branch/hybrid-io/pypy/objspace/std/stringobject.py
   pypy/branch/hybrid-io/pypy/rlib/rgc.py
   pypy/branch/hybrid-io/pypy/rlib/rsocket.py
   pypy/branch/hybrid-io/pypy/rlib/rstring.py
   pypy/branch/hybrid-io/pypy/rlib/rzlib.py
   pypy/branch/hybrid-io/pypy/rlib/streamio.py
   pypy/branch/hybrid-io/pypy/rlib/test/test_rgc.py
   pypy/branch/hybrid-io/pypy/rlib/test/test_rsocket.py
   pypy/branch/hybrid-io/pypy/rlib/test/test_rstring.py
   pypy/branch/hybrid-io/pypy/rpython/annlowlevel.py
   pypy/branch/hybrid-io/pypy/rpython/llinterp.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll2ctypes.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll_str.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llheap.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llmemory.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lloperation.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lltype.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rffi.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rstr.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_llmemory.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_lltype.py
   pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_rffi.py
   pypy/branch/hybrid-io/pypy/rpython/memory/gc/base.py
   pypy/branch/hybrid-io/pypy/rpython/memory/gc/generation.py
   pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/boehm.py
   pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/framework.py
   pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/transform.py
   pypy/branch/hybrid-io/pypy/rpython/memory/gcwrapper.py
   pypy/branch/hybrid-io/pypy/rpython/memory/test/test_gc.py
   pypy/branch/hybrid-io/pypy/rpython/memory/test/test_transformed_gc.py
   pypy/branch/hybrid-io/pypy/rpython/module/ll_os.py
   pypy/branch/hybrid-io/pypy/rpython/module/test/test_posix.py
   pypy/branch/hybrid-io/pypy/rpython/rpbc.py
   pypy/branch/hybrid-io/pypy/rpython/test/test_annlowlevel.py
   pypy/branch/hybrid-io/pypy/rpython/test/test_rbuiltin.py
   pypy/branch/hybrid-io/pypy/rpython/test/test_rlist.py
   pypy/branch/hybrid-io/pypy/rpython/test/test_runicode.py
   pypy/branch/hybrid-io/pypy/rpython/typesystem.py
   pypy/branch/hybrid-io/pypy/translator/backendopt/all.py
   pypy/branch/hybrid-io/pypy/translator/c/funcgen.py
   pypy/branch/hybrid-io/pypy/translator/c/test/test_boehm.py
   pypy/branch/hybrid-io/pypy/translator/c/test/test_lltyped.py
   pypy/branch/hybrid-io/pypy/translator/c/test/test_newgc.py
   pypy/branch/hybrid-io/pypy/translator/exceptiontransform.py
   pypy/branch/hybrid-io/pypy/translator/llvm/opwriter.py
Log:
(fijal, arigo, docgok)
svn merge -r 53620:54648 http://codespeak.net/svn/pypy/dist/pypy .
Merge the io-improvements branch into hybrid-io branch.

This branch approaches problem of io (and string operations) bottlenecks
by introducing few gc primitives that can express better certain situations.

* Implement rgc.can_move(ptr) primitive and rgc.malloc_nonmovable
  which respectively tells whether pointer can move and tries to allocate
  area that will never move. Useful for passing into C-level functions.

* Use this interface to speedup os.read, os.write, rsocket.send/recv
  and rzlib

* Various small performance improvements, including streamio readline
  and rstr using raw_memcopy instead of copying by hand

* Implement resizable_buffer interface, which exposes three functions:
  rgc.resizable_buffer_of_shape, rgc.resize_buffer and
  rgc.finish_building_buffer

* Implement rlib.rstring.StringBuilder and UnicodeBuilder, which uses
  resizable_buffer interface for in-place string building.

* Use StringBuilder/UnicodeBuilder for modulo operation on str/unicode
  and for repr(str)

* Implement GC_REALLOC for boehm, using rffi (which is way nicer :)

* Introduce new operation FORCE_CAST, for casting between different
  types on top of llinterp

* Remove coalloc, it speeded up things by accident



Modified: pypy/branch/hybrid-io/pypy/config/translationoption.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/config/translationoption.py	(original)
+++ pypy/branch/hybrid-io/pypy/config/translationoption.py	Sun May 11 12:45:47 2008
@@ -176,9 +176,6 @@
         BoolOption("heap2stack", "Escape analysis and stack allocation",
                    default=False,
                    requires=[("translation.stackless", False)]),
-        BoolOption("coalloc", "Try to replace mallocs by coallocation",
-                   default=False,
-                   suggests=[("translation.gc", "generation")]),
         # control profile based inlining
         StrOption("profile_based_inline",
                   "Use call count profiling to drive inlining"

Modified: pypy/branch/hybrid-io/pypy/interpreter/pyframe.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/interpreter/pyframe.py	(original)
+++ pypy/branch/hybrid-io/pypy/interpreter/pyframe.py	Sun May 11 12:45:47 2008
@@ -9,8 +9,6 @@
 import opcode
 from pypy.rlib.objectmodel import we_are_translated, instantiate
 from pypy.rlib.jit import we_are_jitted, hint
-from pypy.rlib import rstack # for resume points
-
 
 # Define some opcodes used
 g = globals()
@@ -102,6 +100,7 @@
 
     def execute_frame(self):
         """Execute this frame.  Main entry point to the interpreter."""
+        from pypy.rlib import rstack
         executioncontext = self.space.getexecutioncontext()
         executioncontext.enter(self)
         try:

Modified: pypy/branch/hybrid-io/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/hybrid-io/pypy/interpreter/pyopcode.py	Sun May 11 12:45:47 2008
@@ -18,7 +18,6 @@
 from pypy.tool.stdlib_opcode import opcodedesc, HAVE_ARGUMENT
 from pypy.tool.stdlib_opcode import unrolling_opcode_descs
 from pypy.tool.stdlib_opcode import opcode_method_names
-from pypy.rlib import rstack # for resume points
 from pypy.rlib.unroll import unrolling_iterable
 
 def unaryoperation(operationname):
@@ -70,6 +69,8 @@
 
     def dispatch(self, pycode, next_instr, ec):
         # For the sequel, force 'next_instr' to be unsigned for performance
+        from pypy.rlib import rstack # for resume points
+
         next_instr = r_uint(next_instr)
         co_code = pycode.co_code
 
@@ -82,6 +83,8 @@
             return self.popvalue()
 
     def handle_bytecode(self, co_code, next_instr, ec):
+        from pypy.rlib import rstack # for resume points
+
         try:
             next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
             rstack.resume_point("handle_bytecode", self, co_code, ec,
@@ -203,6 +206,8 @@
                 return next_instr
 
             if we_are_translated():
+                from pypy.rlib import rstack # for resume points
+
                 for opdesc in unrolling_opcode_descs:
                     # static checks to skip this whole case if necessary
                     if not opdesc.is_enabled(space):
@@ -850,6 +855,8 @@
                                   f.space.w_None)
                       
     def call_function(f, oparg, w_star=None, w_starstar=None):
+        from pypy.rlib import rstack # for resume points
+    
         n_arguments = oparg & 0xff
         n_keywords = (oparg>>8) & 0xff
         keywords = None
@@ -863,6 +870,8 @@
         f.pushvalue(w_result)
         
     def CALL_FUNCTION(f, oparg, *ignored):
+        from pypy.rlib import rstack # for resume points
+
         # XXX start of hack for performance
         if (oparg >> 8) & 0xff == 0:
             # Only positional arguments

Modified: pypy/branch/hybrid-io/pypy/objspace/std/formatting.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/objspace/std/formatting.py	(original)
+++ pypy/branch/hybrid-io/pypy/objspace/std/formatting.py	Sun May 11 12:45:47 2008
@@ -5,8 +5,7 @@
 from pypy.rlib.rarithmetic import ovfcheck, formatd_overflow, isnan, isinf
 from pypy.interpreter.error import OperationError
 from pypy.tool.sourcetools import func_with_new_name
-from pypy.rlib.debug import check_annotation
-from pypy.rlib.objectmodel import newlist
+from pypy.rlib.rstring import StringBuilder, UnicodeBuilder
 
 class BaseStringFormatter(object):
     def __init__(self, space, values_w, w_valuedict):
@@ -257,7 +256,10 @@
             return result
 
         def format(self):
-            result = newlist(sizehint=100)      # list of characters or unichars
+            if do_unicode:
+                result = UnicodeBuilder()
+            else:
+                result = StringBuilder()
             self.result = result
             while True:
                 # fast path: consume as many characters as possible
@@ -268,9 +270,9 @@
                         break
                     i += 1
                 else:
-                    result += const(fmt[i0:])
-                    break     # end of 'fmt' string 
-                result += const(fmt[i0:i])
+                    result.append_slice(fmt, i0, len(fmt))
+                    break     # end of 'fmt' string
+                result.append_slice(fmt, i0, i)
                 self.fmtpos = i + 1
 
                 # interpret the next formatter
@@ -296,7 +298,7 @@
                     self.unknown_fmtchar()
 
             self.checkconsumed()
-            return result
+            return result.build()
 
         def unknown_fmtchar(self):
             space = self.space
@@ -317,22 +319,28 @@
 
         def std_wp(self, r):
             length = len(r)
+            if do_unicode and isinstance(r, str):
+                # convert string to unicode explicitely here
+                r = unicode(r)
             prec = self.prec
             if prec == -1 and self.width == 0:
                 # fast path
-                self.result += const(r)
+                self.result.append(const(r))
                 return
             if prec >= 0 and prec < length:
                 length = prec   # ignore the end of the string if too long
             result = self.result
             padding = self.width - length
+            if padding < 0:
+                padding = 0
+            assert padding >= 0
             if not self.f_ljust and padding > 0:
-                result += const(' ') * padding
+                result.append_multiple_char(const(' '), padding)
                 # add any padding at the left of 'r'
                 padding = 0
-            result += const(r[:length])       # add 'r' itself
+            result.append_slice(r, 0, length)       # add 'r' itself
             if padding > 0:
-                result += const(' ') * padding
+                result.append_multiple_char(const(' '), padding)
             # add any remaining padding at the right
         std_wp._annspecialcase_ = 'specialize:argtype(1)'
 
@@ -351,6 +359,8 @@
             # by pushing the pad character into self.result
             result = self.result
             padding = self.width - len(r) - len(prefix)
+            if padding <= 0:
+                padding = 0
 
             if self.f_ljust:
                 padnumber = '<'
@@ -359,16 +369,20 @@
             else:
                 padnumber = '>'
 
+            assert padding >= 0
             if padnumber == '>':
-                result += const(' ') * padding    # pad with spaces on the left
+                result.append_multiple_char(const(' '), padding)
+                # pad with spaces on the left
             if sign:
-                result += const(r[0])        # the sign
-            result += const(prefix)               # the prefix
+                result.append(const(r[0]))        # the sign
+            result.append(const(prefix))               # the prefix
             if padnumber == '0':
-                result += const('0') * padding    # pad with zeroes
-            result += const(r[int(sign):])        # the rest of the number
+                result.append_multiple_char(const('0'), padding)
+                # pad with zeroes
+            result.append_slice(const(r), int(sign), len(r))
+            # the rest of the number
             if padnumber == '<':           # spaces on the right
-                result += const(' ') * padding
+                result.append_multiple_char(const(' '), padding)
 
         def fmt_s(self, w_value):
             space = self.space
@@ -455,14 +469,12 @@
             # fall through to the unicode case
             fmt = unicode(fmt)
         else:
-            check_annotation(result, is_list_of_chars_or_unichars)
-            return space.wrap(''.join(result))
+            return space.wrap(result)
     else:
         fmt = space.unicode_w(w_fmt)
     formatter = UnicodeFormatter(space, fmt, values_w, w_valuedict)
     result = formatter.format()
-    check_annotation(result, is_list_of_chars_or_unichars)
-    return space.wrap(u''.join(result))
+    return space.wrap(result)
 
 def mod_format(space, w_format, w_values, do_unicode=False):
     if space.is_true(space.isinstance(w_values, space.w_tuple)):

Modified: pypy/branch/hybrid-io/pypy/objspace/std/stringobject.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/objspace/std/stringobject.py	(original)
+++ pypy/branch/hybrid-io/pypy/objspace/std/stringobject.py	Sun May 11 12:45:47 2008
@@ -10,6 +10,7 @@
 from pypy.objspace.std.listobject import W_ListObject
 from pypy.objspace.std.noneobject import W_NoneObject
 from pypy.objspace.std.tupleobject import W_TupleObject
+from pypy.rlib.rstring import StringBuilder
 
 from pypy.objspace.std.stringtype import sliced, joined, wrapstr, wrapchar, \
      stringendswith, stringstartswith, joined2
@@ -860,17 +861,17 @@
 def repr__String(space, w_str):
     s = w_str._value
 
-    i = 0
-    buf = [' '] * (len(s) * 4 + 2) # safely overallocate
+    buf = StringBuilder(len(s) + 10)
 
     quote = "'"
     if quote in s and '"' not in s:
         quote = '"'
 
-    buf[i] = quote
+    buf.append(quote)
+    startslice = 0
 
-    for c in s:
-        i += 1
+    for i in range(len(s)):
+        c = s[i]
         use_bs_char = False # character quoted by backspace
 
         if c == '\\' or c == quote:
@@ -887,25 +888,26 @@
             use_bs_char = True
         elif not '\x20' <= c < '\x7f':
             n = ord(c)
-            buf[i] = '\\'
-            i += 1
-            buf[i] = 'x'
-            i += 1
-            buf[i] = "0123456789abcdef"[n>>4]
-            i += 1
-            buf[i] = "0123456789abcdef"[n&0xF]
-        else:
-            buf[i] = c
+            if i != startslice:
+                buf.append_slice(s, startslice, i)
+            startslice = i + 1
+            buf.append('\\x')
+            buf.append("0123456789abcdef"[n>>4])
+            buf.append("0123456789abcdef"[n&0xF])
 
         if use_bs_char:
-            buf[i] = '\\'
-            i += 1
-            buf[i] = bs_char
+            if i != startslice:
+                buf.append_slice(s, startslice, i)
+            startslice = i + 1
+            buf.append('\\')
+            buf.append(bs_char)
+
+    if len(s) != startslice:
+        buf.append_slice(s, startslice, len(s))
 
-    i += 1
-    buf[i] = quote
+    buf.append(quote)
 
-    return space.wrap("".join(buf[:i+1])) # buffer was overallocated, so slice
+    return space.wrap(buf.build())
 
    
 def str_translate__String_ANY_ANY(space, w_string, w_table, w_deletechars=''):

Modified: pypy/branch/hybrid-io/pypy/rlib/rgc.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/rgc.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/rgc.py	Sun May 11 12:45:47 2008
@@ -185,3 +185,139 @@
         opname = 'gc__' + self.instance.__name__
         hop.exception_cannot_occur()
         return hop.genop(opname, [], resulttype=hop.r_result)
+
+def can_move(p):
+    return True
+
+class CanMoveEntry(ExtRegistryEntry):
+    _about_ = can_move
+
+    def compute_result_annotation(self, s_p):
+        from pypy.annotation import model as annmodel
+        return annmodel.SomeBool()
+
+    def specialize_call(self, hop):
+        from pypy.rpython.lltypesystem import lltype
+        hop.exception_cannot_occur()
+        return hop.genop('gc_can_move', hop.args_v, resulttype=hop.r_result)
+
+def malloc_nonmovable(TP, n=None):
+    """ Allocate a non-moving buffer or return nullptr.
+    When running directly, will pretend that gc is always
+    moving (might be configurable in a future)
+    """
+    from pypy.rpython.lltypesystem import lltype
+    return lltype.nullptr(TP)
+
+class MallocNonMovingEntry(ExtRegistryEntry):
+    _about_ = malloc_nonmovable
+
+    def compute_result_annotation(self, s_TP, s_n=None):
+        # basically return the same as malloc
+        from pypy.annotation.builtin import malloc
+        return malloc(s_TP, s_n)
+
+    def specialize_call(self, hop):
+        from pypy.rpython.lltypesystem import lltype
+        # XXX assume flavor and zero to be None by now
+        assert hop.args_s[0].is_constant()
+        vlist = [hop.inputarg(lltype.Void, arg=0)]
+        opname = 'malloc_nonmovable'
+        flags = {'flavor': 'gc'}
+        vlist.append(hop.inputconst(lltype.Void, flags))
+
+        if hop.nb_args == 2:
+            vlist.append(hop.inputarg(lltype.Signed, arg=1))
+            opname += '_varsize'
+
+        hop.exception_cannot_occur()
+        return hop.genop(opname, vlist, resulttype = hop.r_result.lowleveltype)
+
+def resizable_buffer_of_shape(T, init_size):
+    """ Pre-allocates structure of type T (varsized) with possibility
+    to reallocate it further by resize_buffer.
+    """
+    from pypy.rpython.lltypesystem import lltype
+    return lltype.malloc(T, init_size)
+
+class ResizableBufferOfShapeEntry(ExtRegistryEntry):
+    _about_ = resizable_buffer_of_shape
+
+    def compute_result_annotation(self, s_T, s_init_size):
+        from pypy.annotation import model as annmodel
+        from pypy.rpython.lltypesystem import rffi, lltype
+        assert s_T.is_constant()
+        assert isinstance(s_init_size, annmodel.SomeInteger)
+        T = s_T.const
+        # limit ourselves to structs and to a fact that var-sized element
+        # does not contain pointers.
+        assert isinstance(T, lltype.Struct)
+        assert isinstance(getattr(T, T._arrayfld).OF, lltype.Primitive)
+        return annmodel.SomePtr(lltype.Ptr(T))
+
+    def specialize_call(self, hop):
+        from pypy.rpython.lltypesystem import lltype
+        flags = {'flavor': 'gc'}
+        vlist = [hop.inputarg(lltype.Void, 0),
+                 hop.inputconst(lltype.Void, flags),
+                 hop.inputarg(lltype.Signed, 1)]
+        hop.exception_is_here()
+        return hop.genop('malloc_resizable_buffer', vlist,
+                         resulttype=hop.r_result.lowleveltype)
+
+def resize_buffer(ptr, old_size, new_size):
+    """ Resize raw buffer returned by resizable_buffer_of_shape to new size
+    """
+    from pypy.rpython.lltypesystem import lltype
+    T = lltype.typeOf(ptr).TO
+    arrayfld = T._arrayfld
+    arr = getattr(ptr, arrayfld)
+    # we don't have any realloc on top of cpython
+    new_ptr = lltype.malloc(T, new_size)
+    new_ar = getattr(new_ptr, arrayfld)
+    for i in range(old_size):
+        new_ar[i] = arr[i]
+    return new_ptr
+
+class ResizeBufferEntry(ExtRegistryEntry):
+    _about_ = resize_buffer
+
+    def compute_result_annotation(self, s_ptr, s_old_size, s_new_size):
+        from pypy.annotation import model as annmodel
+        from pypy.rpython.lltypesystem import rffi
+        assert isinstance(s_ptr, annmodel.SomePtr)
+        assert isinstance(s_new_size, annmodel.SomeInteger)
+        assert isinstance(s_old_size, annmodel.SomeInteger)
+        return s_ptr
+
+    def specialize_call(self, hop):
+        from pypy.rpython.lltypesystem import lltype
+        vlist = [hop.inputarg(hop.args_r[0], 0),
+                 hop.inputarg(lltype.Signed, 1),
+                 hop.inputarg(lltype.Signed, 2)]
+        hop.exception_is_here()
+        return hop.genop('resize_buffer', vlist,
+                         resulttype=hop.r_result.lowleveltype)
+
+def finish_building_buffer(ptr, final_size):
+    """ Finish building resizable buffer returned by resizable_buffer_of_shape
+    """
+    return ptr
+
+class FinishBuildingBufferEntry(ExtRegistryEntry):
+    _about_ = finish_building_buffer
+
+    def compute_result_annotation(self, s_arr, s_final_size):
+        from pypy.annotation.model import SomePtr, s_ImpossibleValue,\
+             SomeInteger
+        assert isinstance(s_arr, SomePtr)
+        assert isinstance(s_final_size, SomeInteger)
+        return s_arr
+
+    def specialize_call(self, hop):
+        from pypy.rpython.lltypesystem import lltype
+        vlist = [hop.inputarg(hop.args_r[0], 0),
+                 hop.inputarg(hop.args_r[1], 1)]
+        hop.exception_cannot_occur()
+        return hop.genop('finish_building_buffer', vlist,
+                         resulttype=hop.r_result.lowleveltype)

Modified: pypy/branch/hybrid-io/pypy/rlib/rsocket.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/rsocket.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/rsocket.py	Sun May 11 12:45:47 2008
@@ -814,14 +814,13 @@
         if timeout == 1:
             raise SocketTimeout
         elif timeout == 0:
-            buf = mallocbuf(buffersize)
+            raw_buf, gc_buf = rffi.alloc_buffer(buffersize)
             try:
-                read_bytes = _c.socketrecv(self.fd, buf, buffersize, flags)
+                read_bytes = _c.socketrecv(self.fd, raw_buf, buffersize, flags)
                 if read_bytes >= 0:
-                    assert read_bytes <= buffersize
-                    return ''.join([buf[i] for i in range(read_bytes)])
+                    return rffi.str_from_buffer(raw_buf, gc_buf, buffersize, read_bytes)
             finally:
-                lltype.free(buf, flavor='raw')
+                rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
         raise self.error_handler()
 
     def recvfrom(self, buffersize, flags=0):
@@ -832,11 +831,11 @@
         if timeout == 1:
             raise SocketTimeout
         elif timeout == 0:
-            buf = mallocbuf(buffersize)
+            raw_buf, gc_buf = rffi.alloc_buffer(buffersize)
             try:
                 address, addr_p, addrlen_p = self._addrbuf()
                 try:
-                    read_bytes = _c.recvfrom(self.fd, buf, buffersize, flags,
+                    read_bytes = _c.recvfrom(self.fd, raw_buf, buffersize, flags,
                                              addr_p, addrlen_p)
                     addrlen = rffi.cast(lltype.Signed, addrlen_p[0])
                 finally:
@@ -847,10 +846,10 @@
                         address.addrlen = addrlen
                     else:
                         address = None
-                    data = ''.join([buf[i] for i in range(read_bytes)])
+                    data = rffi.str_from_buffer(raw_buf, gc_buf, buffersize, read_bytes)
                     return (data, address)
             finally:
-                lltype.free(buf, flavor='raw')
+                rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
         raise self.error_handler()
 
     def send_raw(self, dataptr, length, flags=0):
@@ -869,18 +868,18 @@
         """Send a data string to the socket.  For the optional flags
         argument, see the Unix manual.  Return the number of bytes
         sent; this may be less than len(data) if the network is busy."""
-        dataptr = rffi.str2charp(data)
+        dataptr = rffi.get_nonmovingbuffer(data)
         try:
             return self.send_raw(dataptr, len(data), flags)
         finally:
-            rffi.free_charp(dataptr)
+            rffi.free_nonmovingbuffer(data, dataptr)
 
     def sendall(self, data, flags=0):
         """Send a data string to the socket.  For the optional flags
         argument, see the Unix manual.  This calls send() repeatedly
         until all data is sent.  If an error occurs, it's impossible
         to tell how much data has been sent."""
-        dataptr = rffi.str2charp(data)
+        dataptr = rffi.get_nonmovingbuffer(data)
         try:
             remaining = len(data)
             p = dataptr
@@ -889,7 +888,7 @@
                 p = rffi.ptradd(p, res)
                 remaining -= res
         finally:
-            rffi.free_charp(dataptr)
+            rffi.free_nonmovingbuffer(data, dataptr)
 
     def sendto(self, data, flags, address):
         """Like send(data, flags) but allows specifying the destination
@@ -1265,7 +1264,7 @@
             raise RSocketError("unknown address family")
         if len(packed) != srcsize:
             raise ValueError("packed IP wrong length for inet_ntop")
-        srcbuf = rffi.str2charp(packed)
+        srcbuf = rffi.get_nonmovingbuffer(packed)
         try:
             dstbuf = mallocbuf(dstsize)
             try:
@@ -1276,7 +1275,7 @@
             finally:
                 lltype.free(dstbuf, flavor='raw')
         finally:
-            lltype.free(srcbuf, flavor='raw')
+            rffi.free_nonmovingbuffer(packed, srcbuf)
 
 def setdefaulttimeout(timeout):
     if timeout < 0.0:

Modified: pypy/branch/hybrid-io/pypy/rlib/rstring.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/rstring.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/rstring.py	Sun May 11 12:45:47 2008
@@ -1,30 +1,52 @@
 
+""" String builder interface
+"""
+
 from pypy.rpython.extregistry import ExtRegistryEntry
-from pypy.annotation import model as annmodel
+from pypy.rpython.annlowlevel import llhelper
+
+INIT_SIZE = 100 # XXX tweak
+
+class AbstractStringBuilder(object):
+    def __init__(self, init_size=INIT_SIZE):
+        self.l = []
 
-def builder(initial_space=20):
-    return []
+    def append(self, s):
+        self.l.append(s)
 
-class SomeStringBuilder(annmodel.SomeObject):
-    def __init__(self, initial_space=0):
-        self.initial_space = initial_space
+    def append_slice(self, s, start, end):
+        self.l.append(s[start:end])
 
-    def method_append(self, s_item):
-        if not isinstance(s_item, annmodel.SomeString):
-            raise TypeError("Can only append strings or characters to string builder")
-        return annmodel.SomeImpossibleValue()
+    def append_multiple_char(self, c, times):
+        self.l.append(c * times)
 
-    def method_build(self):
-        return annmodel.SomeString()
+class StringBuilder(AbstractStringBuilder):
+    def build(self):
+        return "".join(self.l)
 
-class StringBuilderEntry(ExtRegistryEntry):
-    _about_ = builder
+class UnicodeBuilder(AbstractStringBuilder):
+    def build(self):
+        return u''.join(self.l)
 
-    def compute_result_annotation(self, s_initial_space=None):
-        if s_initial_space is None:
-            initial_space = 0
+class BaseEntry(object):
+    def compute_result_annotation(self, s_init_size=None):
+        from pypy.rpython.rbuilder import SomeStringBuilder, SomeUnicodeBuilder
+        if s_init_size is not None:
+            assert s_init_size.is_constant()
+            init_size = s_init_size.const
         else:
-            assert s_initial_space.is_constant()
-            initial_space = s_initial_space.const
-            assert isinstance(initial_space, int)
-        return SomeStringBuilder(initial_space)
+            init_size = INIT_SIZE
+        if self.use_unicode:
+            return SomeUnicodeBuilder(init_size)
+        return SomeStringBuilder(init_size)
+    
+    def specialize_call(self, hop):
+        return hop.r_result.rtyper_new(hop)
+
+class StringBuilderEntry(BaseEntry, ExtRegistryEntry):
+    _about_ = StringBuilder
+    use_unicode = False
+
+class UnicodeBuilderEntry(BaseEntry, ExtRegistryEntry):
+    _about_ = UnicodeBuilder
+    use_unicode = True

Modified: pypy/branch/hybrid-io/pypy/rlib/rzlib.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/rzlib.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/rzlib.py	Sun May 11 12:45:47 2008
@@ -154,9 +154,11 @@
     Compute the CRC32 checksum of the string, possibly with the given
     start value, and return it as a unsigned 32 bit integer.
     """
-    bytes = rffi.str2charp(string)
-    checksum = _crc32(start, rffi.cast(Bytefp, bytes), len(string))
-    rffi.free_charp(bytes)
+    bytes = rffi.get_nonmovingbuffer(string)
+    try:
+        checksum = _crc32(start, rffi.cast(Bytefp, bytes), len(string))
+    finally:
+        rffi.free_nonmovingbuffer(string, bytes)
     return checksum
 
 
@@ -167,9 +169,11 @@
     Compute the Adler-32 checksum of the string, possibly with the given
     start value, and return it as a unsigned 32 bit integer.
     """
-    bytes = rffi.str2charp(string)
-    checksum = _adler32(start, rffi.cast(Bytefp, bytes), len(string))
-    rffi.free_charp(bytes)
+    bytes = rffi.get_nonmovingbuffer(string)
+    try:
+        checksum = _adler32(start, rffi.cast(Bytefp, bytes), len(string))
+    finally:
+        rffi.free_nonmovingbuffer(string, bytes)
     return checksum
 
 # ____________________________________________________________

Modified: pypy/branch/hybrid-io/pypy/rlib/streamio.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/streamio.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/streamio.py	Sun May 11 12:45:47 2008
@@ -466,14 +466,14 @@
                 self.buf = ""
                 return
             while self.lines:
-                line = self.lines[0]
+                line = self.lines[-1]
                 if offset <= len(line):
                     intoffset = intmask(offset)
                     assert intoffset >= 0
-                    self.lines[0] = line[intoffset:]
+                    self.lines[-1] = line[intoffset:]
                     return
-                offset -= len(self.lines[0]) - 1
-                del self.lines[0]
+                offset -= len(self.lines[-1]) - 1
+                self.lines.pop()
             assert not self.lines
             if offset <= len(self.buf):
                 intoffset = intmask(offset)
@@ -500,6 +500,7 @@
             # Skip relative to EOF by reading and saving only just as
             # much as needed
             intoffset = offset2int(offset)
+            self.lines.reverse()
             data = "\n".join(self.lines + [self.buf])
             total = len(data)
             buffers = [data]
@@ -525,6 +526,7 @@
         raise StreamError("whence should be 0, 1 or 2")
 
     def readall(self):
+        self.lines.reverse()
         self.lines.append(self.buf)
         more = ["\n".join(self.lines)]
         self.lines = []
@@ -543,31 +545,33 @@
         assert n >= 0
         if self.lines:
             # See if this can be satisfied from self.lines[0]
-            line = self.lines[0]
+            line = self.lines[-1]
             if len(line) >= n:
-                self.lines[0] = line[n:]
+                self.lines[-1] = line[n:]
                 return line[:n]
 
             # See if this can be satisfied *without exhausting* self.lines
             k = 0
             i = 0
-            for line in self.lines:
+            lgt = len(self.lines)
+            for linenum in range(lgt-1,-1,-1):
+                line = self.lines[linenum]
                 k += len(line)
                 if k >= n:
-                    lines = self.lines[:i]
-                    data = self.lines[i]
+                    lines = self.lines[linenum + 1:]
+                    data = self.lines[linenum]
                     cutoff = len(data) - (k-n)
                     assert cutoff >= 0
                     lines.append(data[:cutoff])
-                    del self.lines[:i]
-                    self.lines[0] = data[cutoff:]
+                    del self.lines[linenum:]
+                    self.lines.append(data[cutoff:])
                     return "\n".join(lines)
                 k += 1
-                i += 1
 
             # See if this can be satisfied from self.lines plus self.buf
             if k + len(self.buf) >= n:
                 lines = self.lines
+                lines.reverse()
                 self.lines = []
                 cutoff = n - k
                 assert cutoff >= 0
@@ -587,6 +591,7 @@
                 return data[:cutoff]
 
         lines = self.lines
+        lines.reverse()
         self.lines = []
         lines.append(self.buf)
         self.buf = ""
@@ -608,16 +613,39 @@
             more[-1] = data[:cutoff]
         return "".join(more)
 
+    # read_next_bunch is generally this, version below is slightly faster
+    #def _read_next_bunch(self):
+    #    self.lines = self.buf.split("\n")
+    #    self.buf = self.lines.pop()
+    #    self.lines.reverse()
+
+    def _read_next_bunch(self):
+        numlines = self.buf.count("\n")
+        self.lines = [None] * numlines
+        last = -1
+        num = numlines - 1
+        while True:
+            start = last + 1
+            assert start >= 0
+            next = self.buf.find("\n", start)
+            if next == -1:
+                if last != -1:
+                    self.buf = self.buf[start:]
+                break
+            assert next >= 0
+            self.lines[num] = self.buf[start:next]
+            last = next
+            num -= 1
+
     def readline(self):
         if self.lines:
-            return self.lines.pop(0) + "\n"
+            return self.lines.pop() + "\n"
 
         # This block is needed because read() can leave self.buf
         # containing newlines
-        self.lines = self.buf.split("\n")
-        self.buf = self.lines.pop()
+        self._read_next_bunch()
         if self.lines:
-            return self.lines.pop(0) + "\n"
+            return self.lines.pop() + "\n"
 
         if self.buf:
             buf = [self.buf]
@@ -625,10 +653,9 @@
             buf = []
         while 1:
             self.buf = self.do_read(self.bufsize)
-            self.lines = self.buf.split("\n")
-            self.buf = self.lines.pop()
+            self._read_next_bunch()
             if self.lines:
-                buf.append(self.lines.pop(0))
+                buf.append(self.lines.pop())
                 buf.append("\n")
                 break
             if not self.buf:
@@ -639,7 +666,7 @@
 
     def peek(self):
         if self.lines:
-            return self.lines[0] + "\n"
+            return self.lines[-1] + "\n"
         else:
             return self.buf
 

Modified: pypy/branch/hybrid-io/pypy/rlib/test/test_rgc.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/test/test_rgc.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/test/test_rgc.py	Sun May 11 12:45:47 2008
@@ -1,6 +1,8 @@
 from pypy.rpython.test.test_llinterp import gengraph, interpret
+from pypy.rpython.lltypesystem import lltype
 from pypy.rlib import rgc # Force registration of gc.collect
 import gc
+import py
 
 def test_collect():
     def f():
@@ -17,3 +19,33 @@
     
     assert res is None
     
+def test_can_move():
+    T0 = lltype.GcStruct('T')
+    T1 = lltype.GcArray(lltype.Float)
+    def f(i):
+        if i:
+            return rgc.can_move(lltype.malloc(T0))
+        else:
+            return rgc.can_move(lltype.malloc(T1, 1))
+
+    t, typer, graph = gengraph(f, [int])
+    ops = list(graph.iterblockops())
+    res = [op for op in ops if op[1].opname == 'gc_can_move']
+    assert len(res) == 2
+
+    res = interpret(f, [1])
+    
+    assert res == True
+    
+def test_resizable_buffer():
+    from pypy.rpython.lltypesystem.rstr import STR
+    from pypy.rpython.annlowlevel import hlstr
+    
+    def f():
+        ptr = rgc.resizable_buffer_of_shape(STR, 1)
+        ptr.chars[0] = 'a'
+        ptr = rgc.resize_buffer(ptr, 1, 2)
+        ptr.chars[1] = 'b'
+        return hlstr(rgc.finish_building_buffer(ptr, 2))
+
+    assert f() == 'ab'

Modified: pypy/branch/hybrid-io/pypy/rlib/test/test_rsocket.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/test/test_rsocket.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/test/test_rsocket.py	Sun May 11 12:45:47 2008
@@ -1,6 +1,7 @@
 import py, errno, sys
 from pypy.rlib import rsocket
 from pypy.rlib.rsocket import *
+from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 import socket as cpy_socket
 
 # cannot test error codes in Win32 because ll2ctypes doesn't save
@@ -268,12 +269,12 @@
 def test_getaddrinfo_no_reverse_lookup():
     # It seems that getaddrinfo never runs a reverse lookup on Linux.
     # Python2.3 on Windows returns the hostname.
-    lst = getaddrinfo('134.99.112.214', None, flags=AI_NUMERICHOST)
+    lst = getaddrinfo('213.239.226.252', None, flags=AI_NUMERICHOST)
     assert isinstance(lst, list)
     found = False
     for family, socktype, protocol, canonname, addr in lst:
-        assert canonname != 'snake.cs.uni-duesseldorf.de'
-        if addr.get_host() == '134.99.112.214':
+        assert canonname != 'codespeak.net'
+        if addr.get_host() == '213.239.226.252':
             found = True
     assert found, lst
 
@@ -370,7 +371,6 @@
     clientsock.close()
     s.close()
 
-
 class TestTCP:
     PORT = 50007
     HOST = 'localhost'

Modified: pypy/branch/hybrid-io/pypy/rlib/test/test_rstring.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rlib/test/test_rstring.py	(original)
+++ pypy/branch/hybrid-io/pypy/rlib/test/test_rstring.py	Sun May 11 12:45:47 2008
@@ -1,27 +1,20 @@
 
-from pypy.translator.translator import TranslationContext
-from pypy.rlib import rstring
-from pypy.annotation import model as annmodel
+from pypy.rlib.rstring import StringBuilder, UnicodeBuilder
 
-class TestAnnotationStringBuilder:
-    def annotate(self, func, args):
-        t = TranslationContext()
-        res = t.buildannotator().build_types(func, args)
-        return t, res
+def test_string_builder():
+    s = StringBuilder()
+    s.append("a")
+    s.append("abc")
+    s.append("a")
+    s.append_slice("abc", 1, 2)
+    s.append_multiple_char('d', 4)
+    assert s.build() == "aabcabdddd"
 
-    def test_builder(self):
-        def f():
-            return rstring.builder()
-        
-        t, res = self.annotate(f, [])
-        assert isinstance(res, rstring.SomeStringBuilder)
-
-    def test_methods(self):
-        def f(x):
-            b = rstring.builder()
-            for i in range(x):
-                b.append("abc")
-            return b.build()
-
-        t, res = self.annotate(f, [int])
-        assert isinstance(res, annmodel.SomeString)
+def test_unicode_builder():
+    s = UnicodeBuilder()
+    s.append(u'a')
+    s.append(u'abc')
+    s.append_slice(u'abcdef', 1, 2)
+    s.append_multiple_char('d', 4)
+    assert s.build() == 'aabcbdddd'
+    assert isinstance(s.build(), unicode)

Modified: pypy/branch/hybrid-io/pypy/rpython/annlowlevel.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/annlowlevel.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/annlowlevel.py	Sun May 11 12:45:47 2008
@@ -400,6 +400,28 @@
         return hop.genop('same_as', [v_ll_str],
                          resulttype = hop.r_result.lowleveltype)
 
+def llstr(s):
+    from pypy.rpython.lltypesystem.rstr import mallocstr
+    # XXX not sure what to do with ootypesystem
+    ll_s = mallocstr(len(s))
+    for i, c in enumerate(s):
+        ll_s.chars[i] = c
+    return ll_s
+
+class LLStrEntry(extregistry.ExtRegistryEntry):
+    _about_ = llstr
+
+    def compute_result_annotation(self, s_str):
+        from pypy.rpython.lltypesystem.rstr import STR
+        return annmodel.lltype_to_annotation(lltype.Ptr(STR))
+
+    def specialize_call(self, hop):
+        hop.exception_cannot_occur()
+        assert hop.args_r[0].lowleveltype == hop.r_result.lowleveltype
+        v_ll_str, = hop.inputargs(*hop.args_r)
+        return hop.genop('same_as', [v_ll_str],
+                         resulttype = hop.r_result.lowleveltype)        
+
 # ____________________________________________________________
 
 def cast_object_to_ptr(PTR, object):

Modified: pypy/branch/hybrid-io/pypy/rpython/llinterp.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/llinterp.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/llinterp.py	Sun May 11 12:45:47 2008
@@ -673,13 +673,6 @@
             self.llinterpreter.remember_malloc(ptr, self)
         return ptr
 
-    def op_coalloc(self, obj, coallocator, flags):
-        flavor = flags['flavor']
-        assert flavor == "gc"
-        zero = flags.get('zero', False)
-        ptr = self.heap.coalloc(obj, coallocator, zero=zero)
-        return ptr
-
     def op_malloc_varsize(self, obj, flags, size):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
@@ -691,14 +684,27 @@
             return ptr
         except MemoryError:
             self.make_llexception()
-
-    def op_coalloc_varsize(self, obj, coallocator, flags, size):
+            
+    def op_malloc_nonmovable(self, obj, flags):
         flavor = flags['flavor']
+        assert flavor == 'gc'
         zero = flags.get('zero', False)
-        assert flavor == "gc"
+        return self.heap.malloc_nonmovable(obj, zero=zero)
+        
+    def op_malloc_nonmovable_varsize(self, obj, flags, size):
+        flavor = flags['flavor']
+        assert flavor == 'gc'
         zero = flags.get('zero', False)
-        ptr = self.heap.coalloc(obj, coallocator, size, zero=zero)
-        return ptr
+        return self.heap.malloc_nonmovable(obj, size, zero=zero)
+
+    def op_malloc_resizable_buffer(self, obj, flags, size):
+        return self.heap.malloc_resizable_buffer(obj, size)
+
+    def op_resize_buffer(self, obj, old_size, new_size):
+        return self.heap.resize_buffer(obj, old_size, new_size)
+
+    def op_finish_building_buffer(self, obj, size):
+        return self.heap.finish_building_buffer(obj, size)
 
     def op_free(self, obj, flavor):
         assert isinstance(flavor, str)
@@ -716,6 +722,11 @@
                               lltype.ContainerType)
         return getattr(obj, field)
 
+    def op_force_cast(self, RESTYPE, obj):
+        from pypy.rpython.lltypesystem import ll2ctypes
+        return ll2ctypes.force_cast(RESTYPE, obj)
+    op_force_cast.need_result_type = True
+
     def op_cast_int_to_ptr(self, RESTYPE, int1):
         return lltype.cast_int_to_ptr(RESTYPE, int1)
     op_cast_int_to_ptr.need_result_type = True
@@ -759,6 +770,9 @@
     def op_gc__enable_finalizers(self):
         self.heap.enable_finalizers()
 
+    def op_gc_can_move(self, addr):
+        return self.heap.can_move(addr)
+
     def op_gc_free(self, addr):
         # what can you do?
         pass

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll2ctypes.py	Sun May 11 12:45:47 2008
@@ -204,7 +204,11 @@
             return
         # regular case: allocate a new ctypes Structure of the proper type
         cls = get_ctypes_type(STRUCT)
-        cstruct = cls._malloc()
+        if STRUCT._arrayfld:
+            n = len(getattr(container, STRUCT._arrayfld).items)
+        else:
+            n = None
+        cstruct = cls._malloc(n)
     add_storage(container, _struct_mixin, cstruct)
     for field_name in STRUCT._names:
         FIELDTYPE = getattr(STRUCT, field_name)
@@ -218,7 +222,9 @@
                 csubstruct = getattr(cstruct, field_name)
                 convert_struct(field_value, csubstruct)
             else:
-                raise NotImplementedError('inlined field', FIELDTYPE)
+                csubarray = getattr(cstruct, field_name)
+                convert_array(field_value, csubarray)
+                #raise NotImplementedError('inlined field', FIELDTYPE)
     remove_regular_struct_content(container)
 
 def remove_regular_struct_content(container):
@@ -228,15 +234,19 @@
         if not isinstance(FIELDTYPE, lltype.ContainerType):
             delattr(container, field_name)
 
-def convert_array(container):
+def convert_array(container, carray=None):
     ARRAY = container._TYPE
     cls = get_ctypes_type(ARRAY)
-    carray = cls._malloc(container.getlength())
+    if carray is None:
+        carray = cls._malloc(container.getlength())
     add_storage(container, _array_mixin, carray)
     if not isinstance(ARRAY.OF, lltype.ContainerType):
+        # fish that we have enough space
+        ctypes_array = ctypes.cast(carray.items,
+                                   ctypes.POINTER(carray.items._type_))
         for i in range(container.getlength()):
             item_value = container.items[i]    # fish fish
-            carray.items[i] = lltype2ctypes(item_value)
+            ctypes_array[i] = lltype2ctypes(item_value)
         remove_regular_array_content(container)
     else:
         assert isinstance(ARRAY.OF, lltype.Struct)
@@ -255,7 +265,10 @@
     remove_regular_struct_content(container)
     for field_name in STRUCT._names:
         FIELDTYPE = getattr(STRUCT, field_name)
-        if isinstance(FIELDTYPE, lltype.ContainerType):
+        if isinstance(FIELDTYPE, lltype.Array):
+            convert_array(getattr(container, field_name),
+                          getattr(ctypes_storage, field_name))
+        elif isinstance(FIELDTYPE, lltype.ContainerType):
             struct_use_ctypes_storage(getattr(container, field_name),
                                       getattr(ctypes_storage, field_name))
 
@@ -375,6 +388,22 @@
 
 # ____________________________________________________________
 
+def _find_parent(llobj):
+    parent, parentindex = lltype.parentlink(llobj)
+    if parent is None:
+        return llobj, 0
+    next_p, next_i = _find_parent(parent)
+    if isinstance(parentindex, int):
+        c_tp = get_ctypes_type(lltype.typeOf(parent))
+        sizeof = ctypes.sizeof(get_ctypes_type(lltype.typeOf(parent).OF))
+        ofs = c_tp.items.offset + parentindex * sizeof
+        return next_p, next_i + ofs
+    else:
+        c_tp = get_ctypes_type(lltype.typeOf(parent))
+        ofs = getattr(c_tp, parentindex).offset
+        return next_p, next_i + ofs
+
+
 # XXX THIS IS A HACK XXX
 # ctypes does not keep callback arguments alive. So we do. Forever
 # we need to think deeper how to approach this problem
@@ -418,6 +447,11 @@
         if T.TO._gckind != 'raw' and not T.TO._hints.get('callback', None):
             raise Exception("can only pass 'raw' data structures to C, not %r"
                             % (T.TO._gckind,))
+
+        index = 0
+        if isinstance(container, lltype._subarray):
+            topmost, index = _find_parent(container)
+            container = topmost
         if container._storage is None:
             raise RuntimeError("attempting to pass a freed structure to C")
         if container._storage is True:
@@ -434,7 +468,17 @@
             container._ctypes_storage_was_allocated()
         storage = container._storage
         p = ctypes.pointer(storage)
-        if normalize and hasattr(storage, '_normalized_ctype'):
+        if index:
+            p = ctypes.cast(p, ctypes.c_void_p)
+            p = ctypes.c_void_p(p.value + index)
+            c_tp = get_ctypes_type(T.TO)
+            storage._normalized_ctype = c_tp
+        if normalize and getattr(T.TO, '_arrayfld', None):
+            # XXX doesn't cache
+            c_tp = build_ctypes_struct(T.TO, [],
+                         len(getattr(storage, T.TO._arrayfld).items))
+            p = ctypes.cast(p, ctypes.POINTER(c_tp))
+        elif normalize and hasattr(storage, '_normalized_ctype'):
             p = ctypes.cast(p, ctypes.POINTER(storage._normalized_ctype))
         return p
 
@@ -449,7 +493,6 @@
 
     if T is lltype.SingleFloat:
         return ctypes.c_float(float(llobj))
-
     return llobj
 
 def ctypes2lltype(T, cobj):
@@ -461,8 +504,10 @@
             return lltype.nullptr(T.TO)
         if isinstance(T.TO, lltype.Struct):
             if T.TO._arrayfld is not None:
-                raise NotImplementedError("XXX var-sized structs")
-            container = lltype._struct(T.TO)
+                lgt = getattr(cobj.contents, T.TO._arrayfld).length
+                container = lltype._struct(T.TO, lgt)
+            else:
+                container = lltype._struct(T.TO)
             struct_use_ctypes_storage(container, cobj.contents)
         elif isinstance(T.TO, lltype.Array):
             if T.TO._hints.get('nolength', False):
@@ -630,6 +675,8 @@
     """Cast a value to a result type, trying to use the same rules as C."""
     if not isinstance(RESTYPE, lltype.LowLevelType):
         raise TypeError("rffi.cast() first arg should be a TYPE")
+    if isinstance(value, llmemory.fakeaddress):
+        value = value.ptr
     TYPE1 = lltype.typeOf(value)
     cvalue = lltype2ctypes(value)
     cresulttype = get_ctypes_type(RESTYPE)
@@ -665,14 +712,12 @@
         return annmodel.lltype_to_annotation(RESTYPE)
 
     def specialize_call(self, hop):
-        from pypy.rpython.rbuiltin import gen_cast
         hop.exception_cannot_occur()
         s_RESTYPE = hop.args_s[0]
         assert s_RESTYPE.is_constant()
         RESTYPE = s_RESTYPE.const
         v_arg = hop.inputarg(hop.args_r[1], arg=1)
-        TYPE1 = v_arg.concretetype
-        return gen_cast(hop.llops, RESTYPE, v_arg)
+        return hop.genop('force_cast', [v_arg], resulttype = RESTYPE)
 
 def typecheck_ptradd(T):
     # --- ptradd() is only for pointers to non-GC, no-length arrays.

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll_str.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll_str.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/ll_str.py	Sun May 11 12:45:47 2008
@@ -1,4 +1,5 @@
 from pypy.rpython.lltypesystem.lltype import GcArray, Array, Char, malloc
+from pypy.rpython.annlowlevel import llstr
 from pypy.rlib.rarithmetic import r_uint, formatd
 
 CHAR_ARRAY = GcArray(Char)
@@ -115,5 +116,5 @@
     return result
 
 def ll_float_str(repr, f):
-    return formatd("%f", f)
+    return llstr(formatd("%f", f))
 

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llheap.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llheap.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llheap.py	Sun May 11 12:45:47 2008
@@ -6,6 +6,7 @@
 setfield = setattr
 from operator import setitem as setarrayitem
 from pypy.rlib.rgc import collect, disable_finalizers, enable_finalizers
+from pypy.rlib.rgc import can_move
 
 def setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue):
     assert typeOf(newvalue) == INNERTYPE
@@ -17,6 +18,19 @@
 def weakref_create_getlazy(objgetter):
     return weakref_create(objgetter())
 
-def coalloc(T, coallocator, n=None, zero=True):
-    # ignore the coallocator
-    return malloc(T, n, zero=zero)
+malloc_nonmovable = malloc
+
+def malloc_resizable_buffer(TP, size):
+    return malloc(TP, size)
+
+def resize_buffer(buf, old_size, new_size):
+    ll_str = malloc(typeOf(buf).TO, new_size)
+    for i in range(old_size):
+        ll_str.chars[i] = buf.chars[i]
+    return ll_str
+
+def finish_building_buffer(buf, final_size):
+    ll_str = malloc(typeOf(buf).TO, final_size)
+    for i in range(final_size):
+        ll_str.chars[i] = buf.chars[i]
+    return ll_str

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llmemory.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llmemory.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/llmemory.py	Sun May 11 12:45:47 2008
@@ -256,6 +256,7 @@
         assert array_type_match(lltype.typeOf(arrayptr).TO, self.TYPE)
         if isinstance(self.TYPE.OF, lltype.ContainerType):
             # XXX this doesn't support empty arrays
+            # XXX it's also missing 'solid' support, probably
             o = arrayptr._obj.getitem(0)
             return o._as_ptr()
         else:

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lloperation.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lloperation.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lloperation.py	Sun May 11 12:45:47 2008
@@ -317,13 +317,17 @@
     'cast_float_to_uint':   LLOp(canfold=True),    # XXX need OverflowError?
     'cast_float_to_longlong':LLOp(canfold=True),
     'truncate_longlong_to_int':LLOp(canfold=True),
+    'force_cast':           LLOp(sideeffects=False),    # only for rffi.cast()
 
     # __________ pointer operations __________
 
     'malloc':               LLOp(canraise=(MemoryError,), canunwindgc=True),
     'malloc_varsize':       LLOp(canraise=(MemoryError,), canunwindgc=True),
-    'coalloc':              LLOp(canraise=(MemoryError,), canunwindgc=True),
-    'coalloc_varsize':      LLOp(canraise=(MemoryError,), canunwindgc=True),
+    'malloc_nonmovable':    LLOp(canraise=(MemoryError,), canunwindgc=True),
+    'malloc_nonmovable_varsize':LLOp(canraise=(MemoryError,),canunwindgc=True),
+    'malloc_resizable_buffer': LLOp(canraise=(MemoryError,),canunwindgc=True),
+    'resize_buffer':        LLOp(canraise=(MemoryError,),canunwindgc=True),
+    'finish_building_buffer' : LLOp(),
     'zero_gc_pointers_inside': LLOp(),
     'free':                 LLOp(),
     'getfield':             LLOp(sideeffects=False, canrun=True),
@@ -399,6 +403,7 @@
     'gc_reload_possibly_moved': LLOp(),
     'gc_id':                LLOp(canraise=(MemoryError,), sideeffects=False),
     'gc_set_max_heap_size': LLOp(),
+    'gc_can_move'         : LLOp(sideeffects=False),
     # experimental operations in support of thread cloning, only
     # implemented by the Mark&Sweep GC
     'gc_x_swap_pool':       LLOp(canraise=(MemoryError,), canunwindgc=True),

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/lltype.py	Sun May 11 12:45:47 2008
@@ -1514,9 +1514,16 @@
     def __init__(self, TYPE, parent, baseoffset_or_fieldname):
         _parentable.__init__(self, TYPE)
         self._setparentstructure(parent, baseoffset_or_fieldname)
+        # Keep the parent array alive, we share the same allocation.
+        # Don't do it if we are inside a GC object, though -- it's someone
+        # else's job to keep the GC object alive
+        if typeOf(top_container(parent))._gckind == 'raw':
+            self._keepparent = parent
 
     def __repr__(self):
-        
+        parent = self._wrparent()
+        if parent is None:
+            return '<_subarray at %s in already freed>' % (self._parent_index,)
         return '<_subarray at %r in %r>' % (self._parent_index,
                                             self._parentstructure(check=False))
 

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rffi.py	Sun May 11 12:45:47 2008
@@ -2,17 +2,23 @@
 from pypy.annotation import model as annmodel
 from pypy.rpython.lltypesystem import lltype
 from pypy.rpython.lltypesystem import ll2ctypes
+from pypy.rpython.lltypesystem.llmemory import cast_adr_to_ptr, cast_ptr_to_adr
+from pypy.rpython.lltypesystem.llmemory import itemoffsetof, offsetof, raw_memcopy
 from pypy.annotation.model import lltype_to_annotation
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import Symbolic, CDefinedIntSymbolic
-from pypy.rlib import rarithmetic
+from pypy.rlib.objectmodel import keepalive_until_here
+from pypy.rlib import rarithmetic, rgc
 from pypy.rpython.extregistry import ExtRegistryEntry
 from pypy.rlib.unroll import unrolling_iterable
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rpython.tool.rfficache import platform
 from pypy.translator.tool.cbuild import ExternalCompilationInfo
 from pypy.translator.backendopt.canraise import RaiseAnalyzer
-from pypy.rpython.annlowlevel import llhelper
+from pypy.rpython.annlowlevel import llhelper, llstr, hlstr
+from pypy.rpython.lltypesystem.rstr import STR
+from pypy.rlib.objectmodel import we_are_translated
+from pypy.rpython.lltypesystem import llmemory
 import os
 
 class UnhandledRPythonException(Exception):
@@ -467,6 +473,95 @@
         l.append(cp[i])
         i += 1
     return "".join(l)
+    
+# str -> char*
+def get_nonmovingbuffer(data):
+    """
+    Either returns a non-moving copy or performs neccessary pointer arithmetic
+    to return a pointer to the characters of a string if the string is already
+    nonmovable.
+    Must be followed by a free_nonmovingbuffer call.
+    """
+    if rgc.can_move(data):
+        count = len(data)
+        buf = lltype.malloc(CCHARP.TO, count, flavor='raw')
+        for i in range(count):
+            buf[i] = data[i]
+        return buf
+    else:
+        data_start = cast_ptr_to_adr(llstr(data)) + \
+            offsetof(STR, 'chars') + itemoffsetof(STR.chars, 0)
+        return cast(CCHARP, data_start)
+
+# (str, char*) -> None
+def free_nonmovingbuffer(data, buf):
+    """
+    Either free a non-moving buffer or keep the original storage alive.
+    """
+    if rgc.can_move(data):
+        lltype.free(buf, flavor='raw')
+    else:
+        keepalive_until_here(data)
+
+# int -> (char*, str)
+def alloc_buffer(count):
+    """
+    Returns a (raw_buffer, gc_buffer) pair, allocated with count bytes.
+    The raw_buffer can be safely passed to a native function which expects it
+    to not move. Call str_from_buffer with the returned values to get a safe
+    high-level string. When the garbage collector cooperates, this allows for
+    the process to be performed without an extra copy.
+    Make sure to call keep_buffer_alive_until_here on the returned values.
+    """
+    str_chars_offset = offsetof(STR, 'chars') + itemoffsetof(STR.chars, 0)
+    gc_buf = rgc.malloc_nonmovable(STR, count)
+    if gc_buf:
+        realbuf = cast_ptr_to_adr(gc_buf) + str_chars_offset
+        raw_buf = cast(CCHARP, realbuf)
+        return raw_buf, gc_buf
+    else:
+        raw_buf = lltype.malloc(CCHARP.TO, count, flavor='raw')
+        return raw_buf, lltype.nullptr(STR)
+alloc_buffer._always_inline_ = True     # to get rid of the returned tuple obj
+
+# (char*, str, int, int) -> None
+def str_from_buffer(raw_buf, gc_buf, allocated_size, needed_size):
+    """
+    Converts from a pair returned by alloc_buffer to a high-level string.
+    The returned string will be truncated to needed_size.
+    """
+    assert allocated_size >= needed_size
+    
+    if gc_buf and (allocated_size == needed_size):
+        return hlstr(gc_buf)
+    
+    new_buf = lltype.malloc(STR, needed_size)
+    try:
+        str_chars_offset = offsetof(STR, 'chars') + itemoffsetof(STR.chars, 0)
+        if gc_buf:
+            src = cast_ptr_to_adr(gc_buf) + str_chars_offset
+        else:
+            src = cast_ptr_to_adr(raw_buf)
+        dest = cast_ptr_to_adr(new_buf) + str_chars_offset
+        ## FIXME: This is bad, because dest could potentially move
+        ## if there are threads involved.
+        raw_memcopy(src, dest,
+                    llmemory.sizeof(lltype.Char) * needed_size)
+        return hlstr(new_buf)
+    finally:
+        keepalive_until_here(new_buf)
+
+# (char*, str) -> None
+def keep_buffer_alive_until_here(raw_buf, gc_buf):
+    """
+    Keeps buffers alive or frees temporary buffers created by alloc_buffer.
+    This must be called after a call to alloc_buffer, usually in a try/finally
+    block.
+    """
+    if gc_buf:
+        keepalive_until_here(gc_buf)
+    elif raw_buf:
+        lltype.free(raw_buf, flavor='raw')
 
 # char* -> str, with an upper bound on the length in case there is no \x00
 def charp2strn(cp, maxlen):

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rstr.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rstr.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/rstr.py	Sun May 11 12:45:47 2008
@@ -15,6 +15,8 @@
      Bool, Void, GcArray, nullptr, pyobjectptr, cast_primitive, typeOf,\
      staticAdtMethod, GcForwardReference
 from pypy.rpython.rmodel import Repr
+from pypy.rpython.lltypesystem import llmemory
+from pypy.tool.sourcetools import func_with_new_name
 
 # ____________________________________________________________
 #
@@ -28,7 +30,7 @@
 STR = GcForwardReference()
 UNICODE = GcForwardReference()
 
-def new_malloc(TP):
+def new_malloc(TP, name):
     def mallocstr(length):
         ll_assert(length >= 0, "negative string length")
         r = malloc(TP, length)
@@ -36,10 +38,10 @@
             r.hash = 0
         return r
     mallocstr._annspecialcase_ = 'specialize:semierased'
-    return mallocstr
+    return func_with_new_name(mallocstr, name)
 
-mallocstr = new_malloc(STR)
-mallocunicode = new_malloc(UNICODE)
+mallocstr = new_malloc(STR, 'mallocstr')
+mallocunicode = new_malloc(UNICODE, 'mallocunicode')
 
 def emptystrfun():
     return emptystr
@@ -47,14 +49,35 @@
 def emptyunicodefun():
     return emptyunicode
 
+def _new_copy_contents_fun(TP, CHAR_TP, name):
+    def _str_ofs(item):
+        return (llmemory.offsetof(TP, 'chars') +
+                llmemory.itemoffsetof(TP.chars, 0) +
+                llmemory.sizeof(CHAR_TP) * item)
+    
+    def copy_string_contents(s1, s2, s1start, s2start, lgt):
+        assert s1start >= 0
+        assert s2start >= 0
+        assert lgt >= 0
+        src = llmemory.cast_ptr_to_adr(s1) + _str_ofs(s1start)
+        dest = llmemory.cast_ptr_to_adr(s2) + _str_ofs(s2start)
+        llmemory.raw_memcopy(src, dest, llmemory.sizeof(CHAR_TP) * lgt)
+    copy_string_contents._always_inline_ = True
+    return func_with_new_name(copy_string_contents, 'copy_%s_contents' % name)
+
+copy_string_contents = _new_copy_contents_fun(STR, Char, 'string')
+copy_unicode_contents = _new_copy_contents_fun(UNICODE, UniChar, 'unicode')
+
 STR.become(GcStruct('rpy_string', ('hash',  Signed),
                     ('chars', Array(Char, hints={'immutable': True})),
                     adtmeths={'malloc' : staticAdtMethod(mallocstr),
-                              'empty'  : staticAdtMethod(emptystrfun)}))
+                              'empty'  : staticAdtMethod(emptystrfun),
+                              'copy_contents' : staticAdtMethod(copy_string_contents)}))
 UNICODE.become(GcStruct('rpy_unicode', ('hash', Signed),
                         ('chars', Array(UniChar, hints={'immutable': True})),
                         adtmeths={'malloc' : staticAdtMethod(mallocunicode),
-                                  'empty'  : staticAdtMethod(emptyunicodefun)}
+                                  'empty'  : staticAdtMethod(emptyunicodefun),
+                                  'copy_contents' : staticAdtMethod(copy_unicode_contents)}
                         ))
 SIGNED_ARRAY = GcArray(Signed)
 CONST_STR_CACHE = WeakValueDictionary()
@@ -217,6 +240,7 @@
             times = 0
         newstr = malloc(times)
         j = 0
+        # XXX we can use memset here, not sure how useful this is
         while j < times:
             newstr.chars[j] = ch
             j += 1
@@ -268,15 +292,8 @@
         len1 = len(s1.chars)
         len2 = len(s2.chars)
         newstr = s1.malloc(len1 + len2)
-        j = 0
-        while j < len1:
-            newstr.chars[j] = s1.chars[j]
-            j += 1
-        i = 0
-        while i < len2:
-            newstr.chars[j] = s2.chars[i]
-            i += 1
-            j += 1
+        s1.copy_contents(s1, newstr, 0, 0, len1)
+        s1.copy_contents(s2, newstr, 0, len1, len2)
         return newstr
 
     def ll_strip(s, ch, left, right):
@@ -293,12 +310,7 @@
                 rpos -= 1
         r_len = rpos - lpos + 1
         result = s.malloc(r_len)
-        i = 0
-        j = lpos
-        while i < r_len:
-            result.chars[i] = s.chars[j]
-            i += 1
-            j += 1
+        s.copy_contents(s, result, lpos, 0, r_len)
         return result
 
     def ll_upper(s):
@@ -307,7 +319,8 @@
         if s_len == 0:
             return s.empty()
         i = 0
-        result = s.malloc(s_len)
+        result = mallocstr(s_len)
+        #        ^^^^^^^^^ specifically to explode on unicode
         while i < s_len:
             ch = s_chars[i]
             if 'a' <= ch <= 'z':
@@ -322,7 +335,8 @@
         if s_len == 0:
             return s.empty()
         i = 0
-        result = s.malloc(s_len)
+        result = mallocstr(s_len)
+        #        ^^^^^^^^^ specifically to explode on unicode
         while i < s_len:
             ch = s_chars[i]
             if 'A' <= ch <= 'Z':
@@ -343,31 +357,15 @@
             itemslen += len(items[i].chars)
             i += 1
         result = s.malloc(itemslen + s_len * (num_items - 1))
-        res_chars = result.chars
-        res_index = 0
-        i = 0
-        item_chars = items[i].chars
-        item_len = len(item_chars)
-        j = 0
-        while j < item_len:
-            res_chars[res_index] = item_chars[j]
-            j += 1
-            res_index += 1
-        i += 1
+        res_index = len(items[0].chars)
+        s.copy_contents(items[0], result, 0, 0, res_index)
+        i = 1
         while i < num_items:
-            j = 0
-            while j < s_len:
-                res_chars[res_index] = s_chars[j]
-                j += 1
-                res_index += 1
-
-            item_chars = items[i].chars
-            item_len = len(item_chars)
-            j = 0
-            while j < item_len:
-                res_chars[res_index] = item_chars[j]
-                j += 1
-                res_index += 1
+            s.copy_contents(s, result, 0, res_index, s_len)
+            res_index += s_len
+            lgt = len(items[i].chars)
+            s.copy_contents(items[i], result, 0, res_index, lgt)
+            res_index += lgt
             i += 1
         return result
 
@@ -599,8 +597,10 @@
             i += 1
         if typeOf(items).TO.OF.TO == STR:
             malloc = mallocstr
+            copy_contents = copy_string_contents
         else:
             malloc = mallocunicode
+            copy_contents = copy_unicode_contents
         result = malloc(itemslen)
         res_chars = result.chars
         res_index = 0
@@ -608,15 +608,14 @@
         while i < num_items:
             item_chars = items[i].chars
             item_len = len(item_chars)
-            j = 0
-            while j < item_len:
-                res_chars[res_index] = item_chars[j]
-                j += 1
-                res_index += 1
+            copy_contents(items[i], result, 0, res_index, item_len)
+            res_index += item_len
             i += 1
         return result
 
     def ll_join_chars(length, chars):
+        # no need to optimize this, will be replaced by string builder
+        # at some point soon
         num_chars = length
         if typeOf(chars).TO.OF == Char:
             malloc = mallocstr
@@ -633,11 +632,10 @@
     def ll_stringslice_startonly(s1, start):
         len1 = len(s1.chars)
         newstr = s1.malloc(len1 - start)
-        j = 0
-        while start < len1:
-            newstr.chars[j] = s1.chars[start]
-            start += 1
-            j += 1
+        lgt = len1 - start
+        assert lgt >= 0
+        assert start >= 0
+        s1.copy_contents(s1, newstr, start, 0, lgt)
         return newstr
 
     def ll_stringslice(s1, slice):
@@ -648,20 +646,17 @@
                 return s1
             stop = len(s1.chars)
         newstr = s1.malloc(stop - start)
-        j = 0
-        while start < stop:
-            newstr.chars[j] = s1.chars[start]
-            start += 1
-            j += 1
+        assert start >= 0
+        lgt = stop - start
+        assert lgt >= 0
+        s1.copy_contents(s1, newstr, start, 0, lgt)
         return newstr
 
     def ll_stringslice_minusone(s1):
         newlen = len(s1.chars) - 1
         newstr = s1.malloc(newlen)
-        j = 0
-        while j < newlen:
-            newstr.chars[j] = s1.chars[j]
-            j += 1
+        assert newlen >= 0
+        s1.copy_contents(s1, newstr, 0, 0, newlen)
         return newstr
 
     def ll_split_chr(LIST, s, c):
@@ -681,22 +676,12 @@
         while j < strlen:
             if chars[j] == c:
                 item = items[resindex] = s.malloc(j - i)
-                newchars = item.chars
-                k = i
-                while k < j:
-                    newchars[k - i] = chars[k]
-                    k += 1
+                item.copy_contents(s, item, i, 0, j - i)
                 resindex += 1
                 i = j + 1
             j += 1
         item = items[resindex] = s.malloc(j - i)
-        newchars = item.chars
-        k = i
-        while k < j:
-            newchars[k - i] = chars[k]
-            k += 1
-        resindex += 1
-
+        item.copy_contents(s, item, i, 0, j - i)
         return res
 
     def ll_replace_chr_chr(s, c1, c2):

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	Sun May 11 12:45:47 2008
@@ -6,11 +6,12 @@
 from pypy.rpython.lltypesystem.ll2ctypes import lltype2ctypes, ctypes2lltype
 from pypy.rpython.lltypesystem.ll2ctypes import standard_c_lib
 from pypy.rpython.lltypesystem.ll2ctypes import uninitialized2ctypes
-from pypy.rpython.lltypesystem.ll2ctypes import ALLOCATED
+from pypy.rpython.lltypesystem.ll2ctypes import ALLOCATED, force_cast
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rlib import rposix
 from pypy.translator.tool.cbuild import ExternalCompilationInfo
 from pypy.tool.udir import udir
+from pypy.rpython.test.test_llinterp import interpret
 
 class TestLL2Ctypes(object):
 
@@ -109,6 +110,19 @@
         lltype.free(a, flavor='raw')
         assert not ALLOCATED     # detects memory leaks in the test
 
+    def test_array_inside_struct(self):
+        # like rstr.STR, but not Gc
+        STR = lltype.Struct('STR', ('x', lltype.Signed), ('y', lltype.Array(lltype.Char)))
+        a = lltype.malloc(STR, 3, flavor='raw')
+        a.y[0] = 'x'
+        a.y[1] = 'y'
+        a.y[2] = 'z'
+        ac = lltype2ctypes(a)
+        assert ac.contents.y.length == 3
+        assert ac.contents.y.items[2] == ord('z')
+        lltype.free(a, flavor='raw')
+        assert not ALLOCATED
+
     def test_array_nolength(self):
         A = lltype.Array(lltype.Signed, hints={'nolength': True})
         a = lltype.malloc(A, 10, flavor='raw')
@@ -357,6 +371,20 @@
         lltype.free(a, flavor='raw')
         assert not ALLOCATED     # detects memory leaks in the test
 
+    def test_adr_cast(self):
+        from pypy.rpython.annlowlevel import llstr
+        from pypy.rpython.lltypesystem.rstr import STR
+        P = lltype.Ptr(lltype.FixedSizeArray(lltype.Char, 1))
+        def f():
+            a = llstr("xyz")
+            b = (llmemory.cast_ptr_to_adr(a) + llmemory.offsetof(STR, 'chars')
+                 + llmemory.itemoffsetof(STR.chars, 0))
+            buf = rffi.cast(rffi.VOIDP, b)
+            return buf[2]
+        assert f() == 'z'
+        res = interpret(f, [])
+        assert res == 'z'
+    
     def test_funcptr1(self):
         def dummy(n):
             return n+1
@@ -789,3 +817,13 @@
         c1 = lltype2ctypes(a1)
         c2 = lltype2ctypes(a2)
         assert type(c1) is type(c2)
+
+    def test_varsized_struct(self): 
+        STR = lltype.Struct('rpy_string', ('hash',  lltype.Signed),
+                            ('chars', lltype.Array(lltype.Char, hints={'immutable': True})))
+        s = lltype.malloc(STR, 3, flavor='raw')
+        one = force_cast(rffi.VOIDP, s)
+        # sanity check
+        #assert lltype2ctypes(one).contents.items._length_ > 0
+        two = force_cast(lltype.Ptr(STR), one)
+        assert s == two

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_llmemory.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_llmemory.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_llmemory.py	Sun May 11 12:45:47 2008
@@ -602,3 +602,13 @@
     py.test.raises(TypeError, "sizeof(S1) <= 0")
     py.test.raises(TypeError, "sizeof(S1) <= 4")
     py.test.raises(TypeError, "(-sizeof(S1)) >= 0")
+
+def test_addr_keeps_object_alive():
+    A = lltype.Array(Address)
+    ptr = lltype.malloc(A, 10, immortal=True)
+    adr = cast_ptr_to_adr(ptr) + ArrayItemsOffset(A)
+    del ptr
+    import gc; gc.collect(); gc.collect()
+    # the following line crashes if the array is dead
+    ptr1 = cast_adr_to_ptr(adr, lltype.Ptr(lltype.FixedSizeArray(Address, 1)))
+    ptr1[0] = NULL

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_lltype.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_lltype.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_lltype.py	Sun May 11 12:45:47 2008
@@ -727,3 +727,11 @@
         setattr(s, word, i)
     for i, word in enumerate(words):
         assert getattr(s, word) == i
+
+def test_subarray_keeps_array_alive():
+    A = Array(Signed)
+    ptr = malloc(A, 10, immortal=True)
+    ptr2 = direct_arrayitems(ptr)
+    del ptr
+    import gc; gc.collect(); gc.collect()
+    ptr2[0] = 5    # crashes if the array was deallocated

Modified: pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_rffi.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_rffi.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/lltypesystem/test/test_rffi.py	Sun May 11 12:45:47 2008
@@ -6,9 +6,11 @@
 from pypy.translator.c.test.test_genc import compile as compile_c
 from pypy.translator.llvm.test.runtest import compile_function as compile_llvm
 from pypy.rpython.lltypesystem.lltype import Signed, Ptr, Char, malloc
+from pypy.rpython.lltypesystem.rstr import STR
 from pypy.rpython.lltypesystem import lltype
 from pypy.tool.udir import udir
 from pypy.rpython.test.test_llinterp import interpret, MallocMismatch
+from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.annotation.annrpython import RPythonAnnotator
 from pypy.rpython.rtyper import RPythonTyper
 from pypy.translator.backendopt.all import backend_optimizations
@@ -429,6 +431,50 @@
         unregister_keepalive(pos, TP)
         assert res == 8
 
+    def test_nonmoving(self):
+        d = 'non-moving data stuff'
+        def f():
+            raw_buf, gc_buf = alloc_buffer(len(d))
+            try:
+                for i in range(len(d)):
+                    raw_buf[i] = d[i]
+                return str_from_buffer(raw_buf, gc_buf, len(d), len(d)-1)
+            finally:
+                keep_buffer_alive_until_here(raw_buf, gc_buf)
+        fn = self.compile(f, [], gcpolicy='ref')
+        assert fn() == d[:-1]
+    
+
+    def test_nonmovingbuffer(self):
+        d = 'some cool data that should not move'
+        def f():
+            buf = get_nonmovingbuffer(d)
+            try:
+                counter = 0
+                for i in range(len(d)):
+                    if buf[i] == d[i]:
+                        counter += 1
+                return counter
+            finally:
+                free_nonmovingbuffer(d, buf)
+        fn = self.compile(f, [], gcpolicy='ref')
+        assert fn() == len(d)
+
+    def test_nonmovingbuffer_semispace(self):
+        d = 'some cool data that should not move'
+        def f():
+            buf = get_nonmovingbuffer(d)
+            try:
+                counter = 0
+                for i in range(len(d)):
+                    if buf[i] == d[i]:
+                        counter += 1
+                return counter
+            finally:
+                free_nonmovingbuffer(d, buf)
+        fn = self.compile(f, [], gcpolicy='semispace')
+        assert fn(expected_extra_mallocs=9) == len(d)
+
 class TestRffiInternals:
     def test_struct_create(self):
         X = CStruct('xx', ('one', INT))
@@ -481,8 +527,7 @@
         graph = graphof(a.translator, f)
         s = summary(graph)
         # there should be not too many operations here by now
-        expected = {'cast_int_to_uint': 1, 'direct_call': 1,
-                    'cast_primitive': 2, 'cast_int_to_float': 1}
+        expected = {'force_cast': 3, 'cast_int_to_float': 1, 'direct_call': 1}
         for k, v in expected.items():
             assert s[k] == v
     
@@ -634,7 +679,6 @@
 def test_ptradd_interpret():
     interpret(test_ptradd, [])
 
-
 class TestCRffi(BaseTestRffi):
     def compile(self, func, args, **kwds):
         return compile_c(func, args, **kwds)
@@ -650,5 +694,13 @@
             del kwds['backendopt']
         return compile_llvm(func, args, **kwds)
 
+    def test_nonmovingbuffer(self):
+        py.test.skip("Somewhat buggy...")
+
+    test_nonmoving = test_nonmovingbuffer
+
+    def test_nonmovingbuffer_semispace(self):
+        py.test.skip("LLVM backend error - unsupported operator")
+
     def test_hashdefine(self):
         py.test.skip("Macros cannot be called as llexternals by design, rffi does not have any special support for them")

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/gc/base.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/gc/base.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/gc/base.py	Sun May 11 12:45:47 2008
@@ -7,6 +7,7 @@
     needs_write_barrier = False
     malloc_zero_filled = False
     prebuilt_gc_objects_are_static_roots = True
+    can_realloc = False
 
     # The following flag enables costly consistency checks after each
     # collection.  It is automatically set to True by test_gc.py.  The
@@ -50,16 +51,13 @@
     def size_gc_header(self, typeid=0):
         return self.gcheaderbuilder.size_gc_header
 
-    def malloc(self, typeid, length=0, zero=False, coallocator=None):
+    def malloc(self, typeid, length=0, zero=False):
         """For testing.  The interface used by the gctransformer is
         the four malloc_[fixed,var]size[_clear]() functions.
-        And (if they exist) to the coalloc_[fixed,var]size_clear functions
         """
         # Rules about fallbacks in case of missing malloc methods:
         #  * malloc_fixedsize_clear() and malloc_varsize_clear() are mandatory
         #  * malloc_fixedsize() and malloc_varsize() fallback to the above
-        #  * coalloc_fixedsize_clear() and coalloc_varsize_clear() are optional
-        # There is no non-clear version of coalloc for now.
         # XXX: as of r49360, gctransformer.framework never inserts calls
         # to malloc_varsize(), but always uses malloc_varsize_clear()
 
@@ -71,39 +69,28 @@
             assert not contains_weakptr
             itemsize = self.varsize_item_sizes(typeid)
             offset_to_length = self.varsize_offset_to_length(typeid)
-            if (coallocator is not None and
-                hasattr(self, "coalloc_varsize_clear")):
-                assert not needs_finalizer
-                coallocator = llmemory.cast_ptr_to_adr(coallocator)
-                ref = self.coalloc_varsize_clear(coallocator, typeid,
-                                                 length, size,
-                                                 itemsize, offset_to_length)
+            if zero or not hasattr(self, 'malloc_varsize'):
+                malloc_varsize = self.malloc_varsize_clear
             else:
-                if zero or not hasattr(self, 'malloc_varsize'):
-                    malloc_varsize = self.malloc_varsize_clear
-                else:
-                    malloc_varsize = self.malloc_varsize
-                ref = malloc_varsize(typeid, length, size, itemsize,
-                                     offset_to_length, True, needs_finalizer)
+                malloc_varsize = self.malloc_varsize
+            ref = malloc_varsize(typeid, length, size, itemsize,
+                                 offset_to_length, True, needs_finalizer)
         else:
-            if (coallocator is not None and
-                hasattr(self, "coalloc_fixedsize_clear")):
-                assert not needs_finalizer
-                coallocator = llmemory.cast_ptr_to_adr(coallocator)
-                ref = self.coalloc_fixedsize_clear(coallocator, typeid, size)
+            if zero or not hasattr(self, 'malloc_fixedsize'):
+                malloc_fixedsize = self.malloc_fixedsize_clear
             else:
-                if zero or not hasattr(self, 'malloc_fixedsize'):
-                    malloc_fixedsize = self.malloc_fixedsize_clear
-                else:
-                    malloc_fixedsize = self.malloc_fixedsize
-                ref = malloc_fixedsize(typeid, size, True, needs_finalizer,
-                                       contains_weakptr)
+                malloc_fixedsize = self.malloc_fixedsize
+            ref = malloc_fixedsize(typeid, size, True, needs_finalizer,
+                                   contains_weakptr)
         # lots of cast and reverse-cast around...
         return llmemory.cast_ptr_to_adr(ref)
 
     def id(self, ptr):
         return lltype.cast_ptr_to_int(ptr)
 
+    def can_move(self, addr):
+        return False
+
     def set_max_heap_size(self, size):
         pass
 
@@ -198,6 +185,8 @@
 class MovingGCBase(GCBase):
     moving_gc = True
 
+    def can_move(self, addr):
+        return True
 
 def choose_gc_from_config(config):
     """Return a (GCClass, GC_PARAMS) from the given config object.

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/gc/generation.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/gc/generation.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/gc/generation.py	Sun May 11 12:45:47 2008
@@ -159,18 +159,6 @@
             self.young_objects_with_weakrefs.append(result + size_gc_header)
         return llmemory.cast_adr_to_ptr(result+size_gc_header, llmemory.GCREF)
 
-    def coalloc_fixedsize_clear(self, coallocator, typeid, size):
-        # note: a coallocated object can never return a weakref, since the
-        # coallocation analysis is done at a time where weakrefs are
-        # represented as opaque objects which aren't allocated using malloc but
-        # with weakref_create
-        if self.is_in_nursery(coallocator):
-            return self.malloc_fixedsize_clear(typeid, size,
-                                               True, False, False)
-        else:
-            return SemiSpaceGC.malloc_fixedsize_clear(self, typeid, size,
-                                                      True, False, False)
-
     def malloc_varsize_clear(self, typeid, length, size, itemsize,
                              offset_to_length, can_collect,
                              has_finalizer=False):
@@ -218,17 +206,6 @@
         self.nursery_free = result + llarena.round_up_for_allocation(totalsize)
         return llmemory.cast_adr_to_ptr(result+size_gc_header, llmemory.GCREF)
 
-    def coalloc_varsize_clear(self, coallocator, typeid,
-                              length, size, itemsize,
-                              offset_to_length):
-        if self.is_in_nursery(coallocator):
-            return self.malloc_varsize_clear(typeid, length, size, itemsize,
-                                             offset_to_length, True, False)
-        else:
-            return SemiSpaceGC.malloc_varsize_clear(self, typeid, length, size,
-                                                    itemsize, offset_to_length,
-                                                    True, False)
-
     # override the init_gc_object methods to change the default value of 'flags',
     # used by objects that are directly created outside the nursery by the SemiSpaceGC.
     # These objects must have the GCFLAG_NO_YOUNG_PTRS flag set immediately.

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/boehm.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/boehm.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/boehm.py	Sun May 11 12:45:47 2008
@@ -4,6 +4,8 @@
 from pypy.rpython.lltypesystem import lltype, llmemory
 from pypy.objspace.flow.model import Constant
 from pypy.rpython.lltypesystem.lloperation import llop
+from pypy.rpython.lltypesystem import rffi
+from pypy.rpython import rmodel
 
 class BoehmGCTransformer(GCTransformer):
     malloc_zero_filled = True
@@ -19,12 +21,19 @@
 
         mh = mallocHelpers()
         mh.allocate = lambda size: llop.boehm_malloc(llmemory.Address, size)
+        c_realloc = rffi.llexternal('GC_REALLOC', [rffi.VOIDP, rffi.INT],
+                                    rffi.VOIDP)
+        def _realloc(ptr, size):
+            return llmemory.cast_ptr_to_adr(c_realloc(rffi.cast(rffi.VOIDP, ptr), size))
+        mh.realloc = _realloc
         ll_malloc_fixedsize = mh._ll_malloc_fixedsize
 
         # XXX, do we need/want an atomic version of this function?
         ll_malloc_varsize_no_length = mh.ll_malloc_varsize_no_length
         ll_malloc_varsize = mh.ll_malloc_varsize
 
+        ll_realloc = mh.ll_realloc
+
         if self.translator:
             self.malloc_fixedsize_ptr = self.inittime_helper(
                 ll_malloc_fixedsize, [lltype.Signed], llmemory.Address)
@@ -39,6 +48,9 @@
                 inline=False)
             self.weakref_deref_ptr = self.inittime_helper(
                 ll_weakref_deref, [llmemory.WeakRefPtr], llmemory.Address)
+            self.realloc_ptr = self.inittime_helper(
+                ll_realloc, [llmemory.Address] + [lltype.Signed] * 4,
+                llmemory.Address)
             self.mixlevelannotator.finish()   # for now
             self.mixlevelannotator.backend_optimize()
 
@@ -48,6 +60,15 @@
     def pop_alive_nopyobj(self, var, llops):
         pass
 
+    def _can_realloc(self):
+        return True
+
+    def perform_realloc(self, hop, v_ptr, v_newlgt, c_const_size, c_item_size,
+                        c_lengthofs):
+        args = [self.realloc_ptr, v_ptr, v_newlgt, c_const_size,
+                c_item_size, c_lengthofs]
+        return hop.genop('direct_call', args, resulttype=llmemory.Address)
+
     def gct_fv_gc_malloc(self, hop, flags, TYPE, c_size):
         # XXX same behavior for zero=True: in theory that's wrong
         if TYPE._is_atomic():
@@ -63,7 +84,6 @@
             hop.genop("boehm_register_finalizer", [v_raw, c_finalizer_ptr])
         return v_raw
 
-
     def gct_fv_gc_malloc_varsize(self, hop, flags, TYPE, v_length, c_const_size, c_item_size,
                                                                    c_offset_to_length):
         # XXX same behavior for zero=True: in theory that's wrong        

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/framework.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/framework.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/framework.py	Sun May 11 12:45:47 2008
@@ -221,6 +221,9 @@
             [s_gc], annmodel.s_None)
         self.enable_finalizers_ptr = getfn(GCClass.enable_finalizers.im_func,
             [s_gc], annmodel.s_None)
+        self.can_move_ptr = getfn(GCClass.can_move.im_func,
+                                  [s_gc, annmodel.SomeAddress()],
+                                  annmodel.SomeBool())
 
         # in some GCs we can inline the common case of
         # malloc_fixedsize(typeid, size, True, False, False)
@@ -292,20 +295,6 @@
                                            inline=True)
         else:
             self.write_barrier_ptr = None
-        if hasattr(GCClass, "coalloc_fixedsize_clear"):
-            self.coalloc_clear_ptr = getfn(
-                GCClass.coalloc_fixedsize_clear.im_func,
-                [s_gc, annmodel.SomeAddress(),
-                 annmodel.SomeInteger(nonneg=True),
-                 annmodel.SomeInteger(nonneg=True)],
-                s_gcref, inline=True)
-            self.coalloc_varsize_clear_ptr = getfn(
-                GCClass.coalloc_varsize_clear.im_func,
-                [s_gc, annmodel.SomeAddress()] +
-                [annmodel.SomeInteger(nonneg=True) for i in range(5)],
-                s_gcref, inline=True)
-        else:
-            self.coalloc_clear_ptr = self.coalloc_varsize_clear_ptr = None
         self.statistics_ptr = getfn(GCClass.statistics.im_func,
                                     [s_gc, annmodel.SomeInteger()],
                                     annmodel.SomeInteger())
@@ -450,7 +439,7 @@
         has_finalizer = bool(self.finalizer_funcptr_for_type(TYPE))
         c_has_finalizer = rmodel.inputconst(lltype.Bool, has_finalizer)
 
-        if not op.opname.endswith('_varsize'):
+        if not op.opname.endswith('_varsize') and not flags.get('varsize'):
             #malloc_ptr = self.malloc_fixedsize_ptr
             zero = flags.get('zero', False)
             if (self.malloc_fast_ptr is not None and
@@ -483,43 +472,6 @@
 
     gct_fv_gc_malloc_varsize = gct_fv_gc_malloc
 
-    def gct_fv_gc_coalloc(self, hop, coallocator, flags, TYPE, *args):
-        if self.coalloc_clear_ptr is None:
-            return self.gct_fv_gc_malloc(
-                    hop, flags, TYPE, *args)
-        op = hop.spaceop
-        flavor = flags['flavor']
-        assert not flags.get("nocollect", False)
-
-        PTRTYPE = op.result.concretetype
-        assert PTRTYPE.TO == TYPE
-        type_id = self.get_type_id(TYPE)
-
-        c_type_id = rmodel.inputconst(lltype.Signed, type_id)
-        info = self.layoutbuilder.type_info_list[type_id]
-        c_size = rmodel.inputconst(lltype.Signed, info.fixedsize)
-        has_finalizer = bool(self.finalizer_funcptr_for_type(TYPE))
-        assert not has_finalizer
-
-        v_coallocator = gen_cast(hop.llops, llmemory.Address, coallocator)
-
-        if not op.opname.endswith('_varsize'):
-            malloc_ptr = self.coalloc_clear_ptr
-            args = [self.c_const_gc, v_coallocator, c_type_id, c_size]
-        else:
-            v_length = op.args[-1]
-            c_ofstolength = rmodel.inputconst(lltype.Signed, info.ofstolength)
-            c_varitemsize = rmodel.inputconst(lltype.Signed, info.varitemsize)
-            malloc_ptr = self.coalloc_varsize_clear_ptr
-            args = [self.c_const_gc, v_coallocator, c_type_id, v_length, c_size,
-                    c_varitemsize, c_ofstolength]
-        livevars = self.push_roots(hop)
-        v_result = hop.genop("direct_call", [malloc_ptr] + args,
-                             resulttype=llmemory.GCREF)
-        self.pop_roots(hop, livevars)
-        return v_result
-    gct_fv_gc_coalloc_varsize = gct_fv_gc_coalloc
-
     def gct_gc__collect(self, hop):
         op = hop.spaceop
         livevars = self.push_roots(hop)
@@ -527,6 +479,16 @@
                   resultvar=op.result)
         self.pop_roots(hop, livevars)
 
+    def gct_gc_can_move(self, hop):
+        op = hop.spaceop
+        v_addr = hop.genop('cast_ptr_to_adr',
+                           [op.args[0]], resulttype=llmemory.Address)
+        hop.genop("direct_call", [self.can_move_ptr, self.c_const_gc, v_addr],
+                  resultvar=op.result)
+
+    def _can_realloc(self):
+        return self.gcdata.gc.can_realloc
+
     def gct_gc__disable_finalizers(self, hop):
         # cannot collect()
         op = hop.spaceop
@@ -629,6 +591,22 @@
                                   self.c_const_gc,
                                   v_size])
 
+    def gct_malloc_nonmovable_varsize(self, hop):
+        TYPE = hop.spaceop.result.concretetype
+        if self.gcdata.gc.moving_gc:
+            # first approximation
+            c = rmodel.inputconst(TYPE, lltype.nullptr(TYPE.TO))
+            return hop.cast_result(c)
+        return self.gct_malloc_varsize(hop)
+
+    def gct_malloc_nonmovable(self, hop):
+        TYPE = hop.spaceop.result.concretetype
+        if self.gcdata.gc.moving_gc:
+            # first approximation
+            c = rmodel.inputconst(TYPE, lltype.nullptr(TYPE.TO))
+            return hop.cast_result(c)
+        return self.gct_malloc(hop)
+
     def transform_generic_set(self, hop):
         from pypy.objspace.flow.model import Constant
         opname = hop.spaceop.opname

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/transform.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/transform.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/gctransform/transform.py	Sun May 11 12:45:47 2008
@@ -455,6 +455,15 @@
         return result
     mh.ll_malloc_varsize_no_length_zero = _ll_malloc_varsize_no_length_zero
 
+    def ll_realloc(ptr, length, constsize, itemsize, lengthoffset):
+        size = constsize + length * itemsize
+        result = mh.realloc(ptr, size)
+        if not result:
+            raise MemoryError()
+        (result + lengthoffset).signed[0] = length
+        return result
+    mh.ll_realloc = ll_realloc
+
     return mh
 
 class GCTransformer(BaseGCTransformer):
@@ -496,22 +505,7 @@
         c_size = rmodel.inputconst(lltype.Signed, llmemory.sizeof(TYPE))
         v_raw = meth(hop, flags, TYPE, c_size)
         hop.cast_result(v_raw)
-    
-    def gct_coalloc(self, hop):
-        TYPE = hop.spaceop.result.concretetype.TO
-        assert not TYPE._is_varsize()
-        flags = hop.spaceop.args[2].value
-        flavor = flags['flavor']
-        c_size = rmodel.inputconst(lltype.Signed, llmemory.sizeof(TYPE))
-        meth = getattr(self, 'gct_fv_%s_comalloc' % flavor, None)
-        if meth is None:
-            meth = getattr(self, 'gct_fv_%s_malloc' % flavor, None)
-            assert meth, "%s has no support for comalloc with flavor %r" % (self, flavor) 
-            v_raw = meth(hop, flags, TYPE, c_size)
-        else:
-            v_raw = meth(hop, hop.spaceop.args[1], flags, TYPE, c_size)
-        hop.cast_result(v_raw)
-
+ 
     def gct_fv_raw_malloc(self, hop, flags, TYPE, c_size):
         v_raw = hop.genop("direct_call", [self.raw_malloc_fixedsize_ptr, c_size],
                           resulttype=llmemory.Address)
@@ -535,19 +529,79 @@
         assert meth, "%s has no support for malloc_varsize with flavor %r" % (self, flavor) 
         return self.varsize_malloc_helper(hop, flags, meth, [])
 
-    def gct_coalloc_varsize(self, hop):
+    gct_malloc_nonmovable = gct_malloc
+    gct_malloc_nonmovable_varsize = gct_malloc_varsize
 
-        flags = hop.spaceop.args[2].value
+    def gct_malloc_resizable_buffer(self, hop):
+        flags = hop.spaceop.args[1].value
+        flags['varsize'] = True
         flavor = flags['flavor']
-        meth = getattr(self, 'gct_fv_%s_coalloc_varsize' % flavor, None)
-        if meth is None:
-            meth = getattr(self, 'gct_fv_%s_malloc_varsize' % flavor, None)
-            assert meth, "%s has no support for malloc_varsize with flavor %r" % (self, flavor) 
-            return self.varsize_malloc_helper(hop, flags, meth, [])
+        assert flavor != 'cpy', "cannot malloc CPython objects directly"
+        meth = getattr(self, 'gct_fv_%s_malloc_varsize' % flavor, None)
+        assert meth, "%s has no support for malloc_varsize with flavor %r" % (self, flavor) 
+        return self.varsize_malloc_helper(hop, flags, meth, [])
+
+    def gct_resize_buffer(self, hop):
+        op = hop.spaceop
+        if self._can_realloc():
+            self._gct_resize_buffer_realloc(hop, op.args[2])
         else:
-            return self.varsize_malloc_helper(hop, flags, meth,
-                                              [hop.spaceop.args[1]])
+            self._gct_resize_buffer_no_realloc(hop, op.args[1])
 
+    def _can_realloc(self):
+        return False
+
+    def _gct_resize_buffer_realloc(self, hop, v_newsize):
+        def intconst(c): return rmodel.inputconst(lltype.Signed, c)
+        op = hop.spaceop
+        flags = {'flavor':'gc', 'varsize': True}
+        TYPE = op.args[0].concretetype.TO
+        ARRAY = TYPE._flds[TYPE._arrayfld]
+        offset_to_length = llmemory.FieldOffset(TYPE, TYPE._arrayfld) + \
+                           llmemory.ArrayLengthOffset(ARRAY)
+        c_const_size = intconst(llmemory.sizeof(TYPE, 0))
+        c_item_size = intconst(llmemory.sizeof(ARRAY.OF))
+
+        c_lengthofs = intconst(offset_to_length)
+        v_ptr = op.args[0]
+        v_raw = self.perform_realloc(hop, v_ptr, v_newsize, c_const_size,
+                                     c_item_size, c_lengthofs)
+        hop.cast_result(v_raw)
+
+    def _gct_resize_buffer_no_realloc(self, hop, v_lgt):
+        op = hop.spaceop
+        meth = self.gct_fv_gc_malloc_varsize
+        flags = {'flavor':'gc', 'varsize': True}
+        self.varsize_malloc_helper(hop, flags, meth, [])
+        # fish resvar
+        v_newbuf = hop.llops[-1].result
+        v_src = op.args[0]
+        TYPE = v_src.concretetype.TO
+        c_fldname = rmodel.inputconst(lltype.Void, TYPE._arrayfld)
+        v_adrsrc = hop.genop('cast_ptr_to_adr', [v_src],
+                             resulttype=llmemory.Address)
+        v_adrnewbuf = hop.genop('cast_ptr_to_adr', [v_newbuf],
+                                resulttype=llmemory.Address)
+        ofs = (llmemory.offsetof(TYPE, TYPE._arrayfld) +
+               llmemory.itemoffsetof(getattr(TYPE, TYPE._arrayfld), 0))
+        v_ofs = rmodel.inputconst(lltype.Signed, ofs)
+        v_adrsrc = hop.genop('adr_add', [v_adrsrc, v_ofs],
+                             resulttype=llmemory.Address)
+        v_adrnewbuf = hop.genop('adr_add', [v_adrnewbuf, v_ofs],
+                                resulttype=llmemory.Address)
+        size = llmemory.sizeof(getattr(TYPE, TYPE._arrayfld).OF)
+        c_size = rmodel.inputconst(lltype.Signed, size)
+        v_lgtsym = hop.genop('int_mul', [c_size, v_lgt],
+                             resulttype=lltype.Signed) 
+        vlist = [v_adrsrc, v_adrnewbuf, v_lgtsym]
+        hop.genop('raw_memcopy', vlist)
+
+    def gct_finish_building_buffer(self, hop):
+        op = hop.spaceop
+        if self._can_realloc():
+            return self._gct_resize_buffer_realloc(hop, op.args[1])
+        else:
+            return self._gct_resize_buffer_no_realloc(hop, op.args[1])
 
     def varsize_malloc_helper(self, hop, flags, meth, extraargs):
         def intconst(c): return rmodel.inputconst(lltype.Signed, c)
@@ -575,10 +629,8 @@
         args = [hop] + extraargs + [flags, TYPE,
                 op.args[-1], c_const_size, c_item_size, c_offset_to_length]
         v_raw = meth(*args)
-
         hop.cast_result(v_raw)
 
-
     def gct_fv_raw_malloc_varsize(self, hop, flags, TYPE, v_length, c_const_size, c_item_size,
                                                                     c_offset_to_length):
         if c_offset_to_length is None:
@@ -608,3 +660,6 @@
             hop.genop('raw_free', [v])
         else:
             assert False, "%s has no support for free with flavor %r" % (self, flavor)           
+
+    def gct_gc_can_move(self, hop):
+        return hop.cast_result(rmodel.inputconst(lltype.Bool, False))

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/gcwrapper.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/gcwrapper.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/gcwrapper.py	Sun May 11 12:45:47 2008
@@ -44,21 +44,42 @@
         else:
             return lltype.malloc(TYPE, n, flavor=flavor, zero=zero)
 
+    def malloc_nonmovable(self, TYPE, n=None, zero=False):
+        typeid = self.get_type_id(TYPE)
+        if self.gc.moving_gc:
+            return lltype.nullptr(TYPE)
+        addr = self.gc.malloc(typeid, n, zero=zero)
+        result = llmemory.cast_adr_to_ptr(addr, lltype.Ptr(TYPE))
+        if not self.gc.malloc_zero_filled:
+            gctypelayout.zero_gc_pointers(result)
+        return result
+
+    def malloc_resizable_buffer(self, TYPE, n):
+        typeid = self.get_type_id(TYPE)
+        addr = self.gc.malloc(typeid, n)
+        result = llmemory.cast_adr_to_ptr(addr, lltype.Ptr(TYPE))
+        if not self.gc.malloc_zero_filled:
+            gctypelayout.zero_gc_pointers(result)
+        return result
+
+    def resize_buffer(self, obj, old_size, new_size):
+        T = lltype.typeOf(obj).TO
+        buf = self.malloc_resizable_buffer(T, new_size)
+        # copy contents
+        arrayfld = T._arrayfld
+        new_arr = getattr(buf, arrayfld)
+        old_arr = getattr(obj, arrayfld)
+        for i in range(old_size):
+            new_arr[i] = old_arr[i]
+        return buf
+
+    def finish_building_buffer(self, obj, size):
+        return obj
+
     def free(self, TYPE, flavor='gc'):
         assert flavor != 'gc'
         return lltype.free(TYPE, flavor=flavor)
 
-    def coalloc(self, TYPE, coallocator, size=None, zero=False):
-        if hasattr(self.gc, "coalloc_fixedsize_clear"):
-            typeid = self.get_type_id(TYPE)
-            addr = self.gc.malloc(typeid, size, zero=zero,
-                                  coallocator=coallocator)
-            result = llmemory.cast_adr_to_ptr(addr, lltype.Ptr(TYPE))
-            if not self.gc.malloc_zero_filled:
-                gctypelayout.zero_gc_pointers(result)
-            return result
-        return self.malloc(TYPE, size, 'gc', zero)
-
     def setfield(self, obj, fieldname, fieldvalue):
         STRUCT = lltype.typeOf(obj).TO
         addr = llmemory.cast_ptr_to_adr(obj)
@@ -89,6 +110,9 @@
     def enable_finalizers(self):
         self.gc.enable_finalizers()
 
+    def can_move(self, addr):
+        return self.gc.can_move(addr)
+
     def weakref_create_getlazy(self, objgetter):
         # we have to be lazy in reading the llinterp variable containing
         # the 'obj' pointer, because the gc.malloc() call below could

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/test/test_gc.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/test/test_gc.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/test/test_gc.py	Sun May 11 12:45:47 2008
@@ -21,6 +21,7 @@
 
 class GCTest(object):
     GC_PARAMS = {}
+    GC_CAN_MOVE = False
 
     def setup_class(cls):
         cls._saved_logstate = py.log._getstate()
@@ -388,12 +389,66 @@
         res = self.interpret(f, [])
         assert res == 42
 
+    def test_can_move(self):
+        TP = lltype.GcArray(lltype.Float)
+        def func():
+            from pypy.rlib import rgc
+            return rgc.can_move(lltype.malloc(TP, 1))
+        assert self.interpret(func, []) == self.GC_CAN_MOVE
+
+    
+    def test_malloc_nonmovable(self):
+        TP = lltype.GcArray(lltype.Char)
+        def func():
+            try:
+                from pypy.rlib import rgc
+                a = rgc.malloc_nonmovable(TP, 3)
+                if a:
+                    assert not rgc.can_move(a)
+                    return 0
+                return 1
+            except Exception, e:
+                return 2
+
+        assert self.interpret(func, []) == int(self.GC_CAN_MOVE)
+
+    def test_malloc_nonmovable_fixsize(self):
+        S = lltype.GcStruct('S', ('x', lltype.Float))
+        TP = lltype.GcStruct('T', ('s', lltype.Ptr(S)))
+        def func():
+            try:
+                from pypy.rlib import rgc
+                a = rgc.malloc_nonmovable(TP)
+                rgc.collect()
+                if a:
+                    assert not rgc.can_move(a)
+                    return 0
+                return 1
+            except Exception, e:
+                return 2
+
+        assert self.interpret(func, []) == int(self.GC_CAN_MOVE)
+
+    def test_resizable_buffer(self):
+        from pypy.rpython.lltypesystem.rstr import STR
+        from pypy.rpython.annlowlevel import hlstr
+        from pypy.rlib import rgc
+
+        def f():
+            ptr = rgc.resizable_buffer_of_shape(STR, 1)
+            ptr.chars[0] = 'a'
+            ptr = rgc.resize_buffer(ptr, 1, 2)
+            ptr.chars[1] = 'b'
+            return len(hlstr(rgc.finish_building_buffer(ptr, 2)))
+
+        assert self.interpret(f, []) == 2
 
 class TestMarkSweepGC(GCTest):
     from pypy.rpython.memory.gc.marksweep import MarkSweepGC as GCClass
 
 class TestSemiSpaceGC(GCTest, snippet.SemiSpaceGCTests):
     from pypy.rpython.memory.gc.semispace import SemiSpaceGC as GCClass
+    GC_CAN_MOVE = True
 
 class TestGrowingSemiSpaceGC(TestSemiSpaceGC):
     GC_PARAMS = {'space_size': 64}
@@ -401,20 +456,6 @@
 class TestGenerationalGC(TestSemiSpaceGC):
     from pypy.rpython.memory.gc.generation import GenerationGC as GCClass
 
-    def test_coalloc(self):
-        def malloc_a_lot():
-            i = 0
-            while i < 10:
-                i += 1
-                a = [1] * 10
-                j = 0
-                while j < 30:
-                    j += 1
-                    a.append(j)
-            return 0
-        res = self.interpret(malloc_a_lot, [], backendopt=True, coalloc=True)
-        assert res == 0
-
 class TestHybridGC(TestGenerationalGC):
     from pypy.rpython.memory.gc.hybrid import HybridGC as GCClass
 

Modified: pypy/branch/hybrid-io/pypy/rpython/memory/test/test_transformed_gc.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/memory/test/test_transformed_gc.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/memory/test/test_transformed_gc.py	Sun May 11 12:45:47 2008
@@ -36,6 +36,7 @@
 class GCTest(object):
     gcpolicy = None
     stacklessgc = False
+    GC_CAN_MOVE = False
 
     def runner(self, f, nbargs=0, statistics=False, transformer=False,
                **extraconfigopts):
@@ -446,8 +447,68 @@
         res = run([])
         assert res == 0
 
+    def test_can_move(self):
+        TP = lltype.GcArray(lltype.Float)
+        def func():
+            from pypy.rlib import rgc
+            return rgc.can_move(lltype.malloc(TP, 1))
+        run = self.runner(func)
+        res = run([])
+        assert res == self.GC_CAN_MOVE
+
+    def test_malloc_nonmovable(self):
+        TP = lltype.GcArray(lltype.Char)
+        def func():
+            try:
+                from pypy.rlib import rgc
+                a = rgc.malloc_nonmovable(TP, 3)
+                rgc.collect()
+                if a:
+                    assert not rgc.can_move(a)
+                    return 0
+                return 1
+            except Exception, e:
+                return 2
+
+        run = self.runner(func)
+        assert int(self.GC_CAN_MOVE) == run([])
+
+    def test_malloc_nonmovable_fixsize(self):
+        S = lltype.GcStruct('S', ('x', lltype.Float))
+        TP = lltype.GcStruct('T', ('s', lltype.Ptr(S)))
+        def func():
+            try:
+                from pypy.rlib import rgc
+                a = rgc.malloc_nonmovable(TP)
+                rgc.collect()
+                if a:
+                    assert not rgc.can_move(a)
+                    return 0
+                return 1
+            except Exception, e:
+                return 2
+
+        run = self.runner(func)
+        assert run([]) == int(self.GC_CAN_MOVE)
+
+    def test_resizable_buffer(self):
+        from pypy.rpython.lltypesystem.rstr import STR
+        from pypy.rpython.annlowlevel import hlstr
+        from pypy.rlib import rgc
+
+        def f():
+            ptr = rgc.resizable_buffer_of_shape(STR, 2)
+            ptr.chars[0] = 'a'
+            ptr = rgc.resize_buffer(ptr, 1, 200)
+            ptr.chars[1] = 'b'
+            return hlstr(rgc.finish_building_buffer(ptr, 2)) == "ab"
+
+        run = self.runner(f)
+        assert run([]) == 1
 
 class GenericMovingGCTests(GenericGCTests):
+    GC_CAN_MOVE = True
+
     def test_many_ids(self):
         py.test.skip("fails for bad reasons in lltype.py :-(")
         class A(object):
@@ -476,7 +537,6 @@
         run = self.runner(f)
         run([])
 
-
 class TestMarkSweepGC(GenericGCTests):
     gcname = "marksweep"
     class gcpolicy(gc.FrameworkGcPolicy):
@@ -699,21 +759,6 @@
         res = run([3, 0])
         assert res == 1
 
-    def test_coalloc(self):
-        def malloc_a_lot():
-            i = 0
-            while i < 10:
-                i += 1
-                a = [1] * 10
-                j = 0
-                while j < 30:
-                    j += 1
-                    a.append(j)
-            return 0
-        run, statistics = self.runner(malloc_a_lot, statistics=True,
-                                      backendopt=True, coalloc=True)
-        run([])
-
 
 class TestPrintingGC(GenericGCTests):
     gcname = "statistics"

Modified: pypy/branch/hybrid-io/pypy/rpython/module/ll_os.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/module/ll_os.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/module/ll_os.py	Sun May 11 12:45:47 2008
@@ -20,10 +20,13 @@
 from pypy.tool.udir import udir
 from pypy.translator.tool.cbuild import ExternalCompilationInfo
 from pypy.rpython.lltypesystem.rstr import mallocstr
-from pypy.rpython.annlowlevel import hlstr
-from pypy.rpython.lltypesystem.llmemory import raw_memcopy, sizeof,\
-     itemoffsetof, cast_ptr_to_adr, offsetof
+from pypy.rpython.annlowlevel import llstr
+from pypy.rpython.lltypesystem.llmemory import sizeof,\
+     itemoffsetof, cast_ptr_to_adr, cast_adr_to_ptr, offsetof
 from pypy.rpython.lltypesystem.rstr import STR
+from pypy.rpython.annlowlevel import llstr
+from pypy.rlib import rgc
+from pypy.rlib.objectmodel import keepalive_until_here
 
 posix = __import__(os.name)
 
@@ -484,20 +487,16 @@
         def os_read_llimpl(fd, count):
             if count < 0:
                 raise OSError(errno.EINVAL, None)
-            inbuf = lltype.malloc(rffi.CCHARP.TO, count, flavor='raw')
+            raw_buf, gc_buf = rffi.alloc_buffer(count)
             try:
-                got = rffi.cast(lltype.Signed, os_read(fd, inbuf, count))
+                void_buf = rffi.cast(rffi.VOIDP, raw_buf)
+                got = rffi.cast(lltype.Signed, os_read(fd, void_buf, count))
                 if got < 0:
                     raise OSError(rposix.get_errno(), "os_read failed")
-                s = mallocstr(got)
-                source = cast_ptr_to_adr(inbuf) + \
-                         itemoffsetof(lltype.typeOf(inbuf).TO, 0)
-                dest = cast_ptr_to_adr(s) + offset
-                raw_memcopy(source, dest, sizeof(lltype.Char) * got)
+                return rffi.str_from_buffer(raw_buf, gc_buf, count, got)
             finally:
-                lltype.free(inbuf, flavor='raw')
-            return hlstr(s)
-
+                rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
+            
         def os_read_oofakeimpl(fd, count):
             return OOSupport.to_rstr(os.read(fd, count))
 
@@ -512,17 +511,15 @@
 
         def os_write_llimpl(fd, data):
             count = len(data)
-            outbuf = lltype.malloc(rffi.CCHARP.TO, count, flavor='raw')
+            buf = rffi.get_nonmovingbuffer(data)
             try:
-                for i in range(count):
-                    outbuf[i] = data[i]
                 written = rffi.cast(lltype.Signed, os_write(
                     rffi.cast(rffi.INT, fd),
-                    outbuf, rffi.cast(rffi.SIZE_T, count)))
+                    buf, rffi.cast(rffi.SIZE_T, count)))
                 if written < 0:
                     raise OSError(rposix.get_errno(), "os_write failed")
             finally:
-                lltype.free(outbuf, flavor='raw')
+                rffi.free_nonmovingbuffer(data, buf)
             return written
 
         def os_write_oofakeimpl(fd, data):

Modified: pypy/branch/hybrid-io/pypy/rpython/module/test/test_posix.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/module/test/test_posix.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/module/test/test_posix.py	Sun May 11 12:45:47 2008
@@ -72,7 +72,10 @@
 
     def test_write(self):
         def f(fi):
-            text = 'This is a test'
+            if fi > 0:
+                text = 'This is a test'
+            else:
+                text = '333'
             return posix.write(fi,text)
         fi = os.open(path,os.O_WRONLY,0777)
         text = 'This is a test'

Modified: pypy/branch/hybrid-io/pypy/rpython/rpbc.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/rpbc.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/rpbc.py	Sun May 11 12:45:47 2008
@@ -11,6 +11,7 @@
         mangle, inputdesc, warning, impossible_repr
 from pypy.rpython import rclass
 from pypy.rpython import robject
+from pypy.rpython.annlowlevel import llstr
 
 from pypy.rpython import callparse
 
@@ -605,6 +606,9 @@
     def none_call(self, hop):
         raise TyperError("attempt to call constant None")
 
+    def ll_str(self, none):
+        return llstr("None")
+
     rtype_simple_call = none_call
     rtype_call_args = none_call
 

Modified: pypy/branch/hybrid-io/pypy/rpython/test/test_annlowlevel.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/test/test_annlowlevel.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/test/test_annlowlevel.py	Sun May 11 12:45:47 2008
@@ -4,7 +4,7 @@
 
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem.rstr import mallocstr
-from pypy.rpython.annlowlevel import hlstr
+from pypy.rpython.annlowlevel import hlstr, llstr
 
 class TestLLType(BaseRtypingTest, LLRtypeMixin):
     def test_hlstr(self):
@@ -13,4 +13,19 @@
         s.chars[1] = "b"
         s.chars[2] = "c"
         assert hlstr(s) == "abc"
+
+    def test_llstr(self):
+        s = llstr("abc")
+        assert len(s.chars) == 3
+        assert s.chars[0] == "a"
+        assert s.chars[1] == "b"
+        assert s.chars[2] == "c"
+
+    def test_llstr_compile(self):
+        def f(arg):
+            s = llstr(hlstr(arg))
+            return len(s.chars)
+
+        res = self.interpret(f, [self.string_to_ll("abc")])
+        assert res == 3
     

Modified: pypy/branch/hybrid-io/pypy/rpython/test/test_rbuiltin.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/test/test_rbuiltin.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/test/test_rbuiltin.py	Sun May 11 12:45:47 2008
@@ -221,10 +221,10 @@
         f = file(tmpfile, 'w')
         f.write('hello world')
         f.close()
-        def f():
+        def fn():
             fd = os.open(tmpfile, os.O_RDONLY, 0777)
             return os.read(fd, 4096)
-        res = self.interpret(f, [])
+        res = self.interpret(fn, [])
         assert self.ll_to_string(res) == 'hello world'
 
     def test_os_lseek(self):

Modified: pypy/branch/hybrid-io/pypy/rpython/test/test_rlist.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/test/test_rlist.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/test/test_rlist.py	Sun May 11 12:45:47 2008
@@ -730,6 +730,12 @@
         res = self.interpret(fn, [])
         assert self.ll_to_string(res) == fn()
 
+        def fn():
+            return str([1.25])
+
+        res = self.interpret(fn, [])
+        assert eval(self.ll_to_string(res)) == [1.25]
+
     def test_list_or_None(self):
         empty_list = []
         nonempty_list = [1, 2]

Modified: pypy/branch/hybrid-io/pypy/rpython/test/test_runicode.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/test/test_runicode.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/test/test_runicode.py	Sun May 11 12:45:47 2008
@@ -211,10 +211,6 @@
     test_float = unsupported
     test_hlstr = unsupported
 
-    def test_find_empty_string(self):
-        py.test.skip("We should think how to solve this problem")
-    test_rfind_empty_string = test_find_empty_string
-
 class TestLLtype(BaseTestRUnicode, LLRtypeMixin):
     EMPTY_STRING_HASH = -1
 

Modified: pypy/branch/hybrid-io/pypy/rpython/typesystem.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/rpython/typesystem.py	(original)
+++ pypy/branch/hybrid-io/pypy/rpython/typesystem.py	Sun May 11 12:45:47 2008
@@ -22,7 +22,7 @@
                 return None
         if name in ('rclass', 'rpbc', 'rbuiltin', 'rtuple', 'rlist',
                     'rslice', 'rdict', 'rrange', 'rstr', 'rgeneric',
-                    'll_str', 'exceptiondata'):
+                    'll_str', 'rbuilder', 'exceptiondata'):
             mod = load(name)
             if mod is not None:
                 setattr(self, name, mod)

Modified: pypy/branch/hybrid-io/pypy/translator/backendopt/all.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/backendopt/all.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/backendopt/all.py	Sun May 11 12:45:47 2008
@@ -130,10 +130,6 @@
         print "after if-to-switch:"
         print_statistics(translator.graphs[0], translator)
 
-    if config.coalloc and not secondary:
-        from pypy.translator.backendopt import coalloc
-        coalloc.malloc_to_coalloc(translator)
-
     remove_obvious_noops()
 
     for graph in graphs:

Modified: pypy/branch/hybrid-io/pypy/translator/c/funcgen.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/c/funcgen.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/c/funcgen.py	Sun May 11 12:45:47 2008
@@ -699,6 +699,8 @@
         typename = cdecl(self.db.gettype(TYPE), '')        
         return "%(result)s = (%(typename)s)(%(val)s);" % locals()
 
+    OP_FORCE_CAST = OP_CAST_PRIMITIVE   # xxx the same logic works
+
     def OP_RESUME_POINT(self, op):
         return '/* resume point %s */'%(op.args[0],)
 

Modified: pypy/branch/hybrid-io/pypy/translator/c/test/test_boehm.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/c/test/test_boehm.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/c/test/test_boehm.py	Sun May 11 12:45:47 2008
@@ -367,6 +367,31 @@
         res = c_fn(10000)
         assert res == 0
 
+    def test_can_move(self):
+        from pypy.rlib import rgc
+        class A:
+            pass
+        def fn():
+            return rgc.can_move(A())
+
+        c_fn = self.getcompiled(fn, [])
+        assert c_fn() == False
+
+    def test_resizable_buffer(self):
+        from pypy.rpython.lltypesystem.rstr import STR
+        from pypy.rpython.annlowlevel import hlstr
+        from pypy.rlib import rgc
+
+        def f():
+            ptr = rgc.resizable_buffer_of_shape(STR, 2)
+            ptr.chars[0] = 'a'
+            ptr = rgc.resize_buffer(ptr, 1, 200)
+            ptr.chars[1] = 'b'
+            return hlstr(rgc.finish_building_buffer(ptr, 2)) == "ab"
+
+        run = self.getcompiled(f)
+        assert run() == True
+
     # reusing some tests from pypy.rpython.memory.test.snippet
     large_tests_ok = True
 

Modified: pypy/branch/hybrid-io/pypy/translator/c/test/test_lltyped.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/c/test/test_lltyped.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/c/test/test_lltyped.py	Sun May 11 12:45:47 2008
@@ -304,6 +304,23 @@
         res = fn(100)
         assert res == 100 + len(list(names))
 
+    def test_force_cast(self):
+        from pypy.rpython.annlowlevel import llstr
+        from pypy.rpython.lltypesystem.rstr import STR
+        from pypy.rpython.lltypesystem import rffi, llmemory, lltype
+        P = lltype.Ptr(lltype.FixedSizeArray(lltype.Char, 1))
+        
+        def f():
+            a = llstr("xyz")
+            b = (llmemory.cast_ptr_to_adr(a) + llmemory.offsetof(STR, 'chars')
+                 + llmemory.itemoffsetof(STR.chars, 0))
+            buf = rffi.cast(rffi.VOIDP, b)
+            return buf[2]
+        
+        fn = self.getcompiled(f, [])
+        res = fn()
+        assert res == 'z'
+
     def test_array_nolength(self):
         A = Array(Signed, hints={'nolength': True})
         a1 = malloc(A, 3, immortal=True)

Modified: pypy/branch/hybrid-io/pypy/translator/c/test/test_newgc.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/c/test/test_newgc.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/c/test/test_newgc.py	Sun May 11 12:45:47 2008
@@ -2,6 +2,7 @@
 import sys
 import py
 from py.test import raises
+import os
 
 from pypy.objspace.flow.model import summary
 from pypy.translator.translator import TranslationContext
@@ -12,6 +13,7 @@
 from pypy.rpython.memory.test import snippet
 from pypy.rlib.objectmodel import keepalive_until_here
 from pypy import conftest
+from pypy.tool.udir import udir
 
 def compile_func(fn, inputtypes, t=None, gcpolicy="ref"):
     from pypy.config.pypyoption import get_pypy_config
@@ -814,6 +816,25 @@
         c_fn = self.getcompiled(f)
         assert c_fn() == 0
 
+    def test_open_read_write_seek_close(self):
+        filename = str(udir.join('test_open_read_write_close.txt'))
+        def does_stuff():
+            fd = os.open(filename, os.O_WRONLY | os.O_CREAT, 0777)
+            count = os.write(fd, "hello world\n")
+            assert count == len("hello world\n")
+            os.close(fd)
+            fd = os.open(filename, os.O_RDONLY, 0777)
+            result = os.lseek(fd, 1, 0)
+            assert result == 1
+            data = os.read(fd, 500)
+            assert data == "ello world\n"
+            os.close(fd)
+
+        f1 = self.getcompiled(does_stuff)
+        f1()
+        assert open(filename, 'r').read() == "hello world\n"
+        os.unlink(filename)
+
     def test_callback_with_collect(self):
         from pypy.rlib.libffi import ffi_type_pointer, cast_type_to_ffitype,\
              CDLL, ffi_type_void, CallbackFuncPtr, ffi_type_sint
@@ -918,7 +939,6 @@
         res = c_fn()
         assert res == 2
 
-
 class TestGenerationalGC(TestSemiSpaceGC):
     gcpolicy = "generation"
     should_be_moving = True

Modified: pypy/branch/hybrid-io/pypy/translator/exceptiontransform.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/exceptiontransform.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/exceptiontransform.py	Sun May 11 12:45:47 2008
@@ -346,7 +346,10 @@
         block.recloseblock(l0, l)
 
         insert_zeroing_op = False
-        if spaceop.opname == 'malloc':
+        # XXX this is not right. it also inserts zero_gc_pointers_inside
+        # XXX on a path that malloc_nonmovable returns null, but does not raise
+        # XXX which might end up with a segfault. But we don't have such gc now
+        if spaceop.opname == 'malloc' or spaceop.opname == 'malloc_nonmovable':
             flavor = spaceop.args[1].value['flavor']
             if flavor == 'gc':
                 insert_zeroing_op = True

Modified: pypy/branch/hybrid-io/pypy/translator/llvm/opwriter.py
==============================================================================
--- pypy/branch/hybrid-io/pypy/translator/llvm/opwriter.py	(original)
+++ pypy/branch/hybrid-io/pypy/translator/llvm/opwriter.py	Sun May 11 12:45:47 2008
@@ -139,6 +139,8 @@
                 self.cast_int_to_ptr(opr)
             else:
                 self.cast_primitive(opr)
+        elif op.opname == 'force_cast':
+            self.cast_primitive(opr)
         else:
             meth = getattr(self, op.opname, None)
             if not meth:



More information about the Pypy-commit mailing list