[Python-checkins] r64232 - in python/branches/tlee-ast-optimize: Doc/includes/mp_benchmarks.py Doc/includes/mp_distributing.py Doc/includes/mp_newtype.py Doc/includes/mp_pool.py Doc/includes/mp_synchronize.py Doc/includes/mp_webserver.py Doc/includes/mp_workers.py Doc/library/ast.rst Doc/library/ctypes.rst Doc/library/multiprocessing.rst Doc/library/numbers.rst Doc/library/someos.rst Doc/library/stdtypes.rst Doc/whatsnew/2.6.rst Include/bytesobject.h Include/pymem.h Include/pyport.h Include/stringobject.h Lib/asyncore.py Lib/multiprocessing Lib/numbers.py Lib/test/test_array.py Lib/test/test_asyncore.py Lib/test/test_heapq.py Lib/test/test_multiprocessing.py Lib/test/test_platform.py Lib/test/test_pydoc.py Lib/test/test_set.py Lib/test/test_struct.py Makefile.pre.in Misc/NEWS Modules/_csv.c Modules/_heapqmodule.c Modules/_multiprocessing Modules/_struct.c Modules/arraymodule.c Modules/audioop.c Modules/binascii.c Modules/cPickle.c Modules/cStringIO.c Modules/cjkcodecs/multibytecodec.c Modules/datetimemodule.c Modules/md5.c Modules/stropmodule.c Objects/bufferobject.c Objects/bytesobject.c Objects/fileobject.c Objects/listobject.c Objects/obmalloc.c Objects/setobject.c Objects/stringobject.c PC/VC6/pythoncore.dsp PC/VS7.1/pythoncore.vcproj PC/VS8.0/pythoncore.vcproj PCbuild/pythoncore.vcproj Parser/node.c Python/asdl.c Python/ast.c Python/bltinmodule.c Python/compile.c setup.py
thomas.lee
python-checkins at python.org
Fri Jun 13 16:43:16 CEST 2008
Author: thomas.lee
Date: Fri Jun 13 16:43:12 2008
New Revision: 64232
Log:
Merged revisions 64073-64120 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r64080 | josiah.carlson | 2008-06-11 01:58:19 +1000 (Wed, 11 Jun 2008) | 3 lines
Fixed test to reflect new filedispatcher semantics, as well as two
NameErrors pointed out by Giampaolo.
........
r64086 | gregory.p.smith | 2008-06-11 03:42:36 +1000 (Wed, 11 Jun 2008) | 5 lines
More reverting of r63675 per the mailing list discussions. This restores
occurances of PyBytes_ in the code to their original PyString_ names. The
bytesobject.c file will be renamed back to stringobject.c in a future checkin.
........
r64089 | armin.ronacher | 2008-06-11 06:37:02 +1000 (Wed, 11 Jun 2008) | 3 lines
Fix a formatting error in the ast documentation.
........
r64090 | armin.ronacher | 2008-06-11 06:52:19 +1000 (Wed, 11 Jun 2008) | 3 lines
Documented the new AST constructor.
........
r64093 | gregory.p.smith | 2008-06-11 07:23:22 +1000 (Wed, 11 Jun 2008) | 3 lines
Rename bytesobject.c back to stringobject.c to keep with the PyString theme.
Part of reverting most of r63675 per the mailing list discussion.
........
r64095 | amaury.forgeotdarc | 2008-06-11 07:37:15 +1000 (Wed, 11 Jun 2008) | 3 lines
Correct test_pydoc for win32 platforms, to account for normalized URLs:
C:\temp => file:///C|temp/
........
r64097 | benjamin.peterson | 2008-06-11 08:39:25 +1000 (Wed, 11 Jun 2008) | 2 lines
backport of 64096
........
r64098 | raymond.hettinger | 2008-06-11 10:25:29 +1000 (Wed, 11 Jun 2008) | 6 lines
Mini-PEP: Simplifying numbers.py
* Convert binary methods in Integral to mixin methods
* Remove three-arg __pow__ as a required method
* Make __int__ the root method instead of __long__.
........
r64100 | raymond.hettinger | 2008-06-11 10:28:51 +1000 (Wed, 11 Jun 2008) | 1 line
Update numbers doc for the Integral simplification.
........
r64101 | raymond.hettinger | 2008-06-11 10:44:47 +1000 (Wed, 11 Jun 2008) | 3 lines
Handle the case with zero arguments.
........
r64102 | benjamin.peterson | 2008-06-11 11:31:28 +1000 (Wed, 11 Jun 2008) | 4 lines
convert test_struct to a unittest thanks to Giampaolo Rodola
I had to disable one test because it was functioning incorrectly, see #1530559
I also removed the debugging prints
........
r64104 | benjamin.peterson | 2008-06-11 12:40:25 +1000 (Wed, 11 Jun 2008) | 2 lines
add the multiprocessing package to fulfill PEP 371
........
r64105 | gregory.p.smith | 2008-06-11 13:40:10 +1000 (Wed, 11 Jun 2008) | 3 lines
swap stringobject.h and bytesobject.h contents to make sense. PyString in
stringobject and PyBytes defines in bytesobject.
........
r64113 | thomas.heller | 2008-06-11 17:10:43 +1000 (Wed, 11 Jun 2008) | 2 lines
Fix markup.
Document the new 'offset' parameter for the 'ctypes.byref' function.
........
r64114 | gregory.p.smith | 2008-06-11 17:41:16 +1000 (Wed, 11 Jun 2008) | 6 lines
Merge in release25-maint r60793:
Added checks for integer overflows, contributed by Google. Some are
only available if asserts are left in the code, in cases where they
can't be triggered from Python code.
........
r64115 | raymond.hettinger | 2008-06-11 20:30:54 +1000 (Wed, 11 Jun 2008) | 1 line
Multi-arg form for set.difference() and set.difference_update().
........
r64116 | raymond.hettinger | 2008-06-11 22:06:49 +1000 (Wed, 11 Jun 2008) | 1 line
Issue 3051: Let heapq work with either __lt__ or __le__.
........
r64117 | benjamin.peterson | 2008-06-11 22:26:31 +1000 (Wed, 11 Jun 2008) | 2 lines
fix import of multiprocessing by juggling imports
........
r64118 | raymond.hettinger | 2008-06-11 22:39:09 +1000 (Wed, 11 Jun 2008) | 1 line
Optimize previous checkin for heapq.
........
r64119 | andrew.kuchling | 2008-06-11 22:53:14 +1000 (Wed, 11 Jun 2008) | 1 line
Note PEP 371 section
........
r64120 | raymond.hettinger | 2008-06-11 23:14:50 +1000 (Wed, 11 Jun 2008) | 1 line
Add test for heapq using both __lt__ and __le__.
........
Added:
python/branches/tlee-ast-optimize/Doc/includes/mp_benchmarks.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_benchmarks.py
python/branches/tlee-ast-optimize/Doc/includes/mp_distributing.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_distributing.py
python/branches/tlee-ast-optimize/Doc/includes/mp_newtype.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_newtype.py
python/branches/tlee-ast-optimize/Doc/includes/mp_pool.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_pool.py
python/branches/tlee-ast-optimize/Doc/includes/mp_synchronize.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_synchronize.py
python/branches/tlee-ast-optimize/Doc/includes/mp_webserver.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_webserver.py
python/branches/tlee-ast-optimize/Doc/includes/mp_workers.py
- copied unchanged from r64120, /python/trunk/Doc/includes/mp_workers.py
python/branches/tlee-ast-optimize/Doc/library/multiprocessing.rst
- copied unchanged from r64120, /python/trunk/Doc/library/multiprocessing.rst
python/branches/tlee-ast-optimize/Lib/multiprocessing/
- copied from r64120, /python/trunk/Lib/multiprocessing/
python/branches/tlee-ast-optimize/Lib/test/test_multiprocessing.py
- copied unchanged from r64120, /python/trunk/Lib/test/test_multiprocessing.py
python/branches/tlee-ast-optimize/Modules/_multiprocessing/
- copied from r64120, /python/trunk/Modules/_multiprocessing/
python/branches/tlee-ast-optimize/Objects/stringobject.c
- copied unchanged from r64120, /python/trunk/Objects/stringobject.c
Removed:
python/branches/tlee-ast-optimize/Objects/bytesobject.c
Modified:
python/branches/tlee-ast-optimize/ (props changed)
python/branches/tlee-ast-optimize/Doc/library/ast.rst
python/branches/tlee-ast-optimize/Doc/library/ctypes.rst
python/branches/tlee-ast-optimize/Doc/library/numbers.rst
python/branches/tlee-ast-optimize/Doc/library/someos.rst
python/branches/tlee-ast-optimize/Doc/library/stdtypes.rst
python/branches/tlee-ast-optimize/Doc/whatsnew/2.6.rst
python/branches/tlee-ast-optimize/Include/bytesobject.h
python/branches/tlee-ast-optimize/Include/pymem.h
python/branches/tlee-ast-optimize/Include/pyport.h
python/branches/tlee-ast-optimize/Include/stringobject.h
python/branches/tlee-ast-optimize/Lib/asyncore.py
python/branches/tlee-ast-optimize/Lib/numbers.py
python/branches/tlee-ast-optimize/Lib/test/test_array.py
python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py
python/branches/tlee-ast-optimize/Lib/test/test_heapq.py
python/branches/tlee-ast-optimize/Lib/test/test_platform.py
python/branches/tlee-ast-optimize/Lib/test/test_pydoc.py
python/branches/tlee-ast-optimize/Lib/test/test_set.py
python/branches/tlee-ast-optimize/Lib/test/test_struct.py
python/branches/tlee-ast-optimize/Makefile.pre.in
python/branches/tlee-ast-optimize/Misc/NEWS
python/branches/tlee-ast-optimize/Modules/_csv.c
python/branches/tlee-ast-optimize/Modules/_heapqmodule.c
python/branches/tlee-ast-optimize/Modules/_struct.c
python/branches/tlee-ast-optimize/Modules/arraymodule.c
python/branches/tlee-ast-optimize/Modules/audioop.c
python/branches/tlee-ast-optimize/Modules/binascii.c
python/branches/tlee-ast-optimize/Modules/cPickle.c
python/branches/tlee-ast-optimize/Modules/cStringIO.c
python/branches/tlee-ast-optimize/Modules/cjkcodecs/multibytecodec.c
python/branches/tlee-ast-optimize/Modules/datetimemodule.c
python/branches/tlee-ast-optimize/Modules/md5.c
python/branches/tlee-ast-optimize/Modules/stropmodule.c
python/branches/tlee-ast-optimize/Objects/bufferobject.c
python/branches/tlee-ast-optimize/Objects/fileobject.c
python/branches/tlee-ast-optimize/Objects/listobject.c
python/branches/tlee-ast-optimize/Objects/obmalloc.c
python/branches/tlee-ast-optimize/Objects/setobject.c
python/branches/tlee-ast-optimize/PC/VC6/pythoncore.dsp
python/branches/tlee-ast-optimize/PC/VS7.1/pythoncore.vcproj
python/branches/tlee-ast-optimize/PC/VS8.0/pythoncore.vcproj
python/branches/tlee-ast-optimize/PCbuild/pythoncore.vcproj
python/branches/tlee-ast-optimize/Parser/node.c
python/branches/tlee-ast-optimize/Python/asdl.c
python/branches/tlee-ast-optimize/Python/ast.c
python/branches/tlee-ast-optimize/Python/bltinmodule.c
python/branches/tlee-ast-optimize/Python/compile.c
python/branches/tlee-ast-optimize/setup.py
Modified: python/branches/tlee-ast-optimize/Doc/library/ast.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/ast.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/ast.rst Fri Jun 13 16:43:12 2008
@@ -96,6 +96,11 @@
node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
lineno=0, col_offset=0)
+ .. versionadded:: 2.6
+ The constructor as explained above was added. In Python 2.5 nodes had
+ to be created by calling the class constructor without arguments and
+ setting the attributes afterwards.
+
.. _abstract-grammar:
@@ -135,7 +140,7 @@
from untrusted sources without the need to parse the values oneself.
-.. function:: get_docstring(node, clean=True):
+.. function:: get_docstring(node, clean=True)
Return the docstring of the given *node* (which must be a
:class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None``
Modified: python/branches/tlee-ast-optimize/Doc/library/ctypes.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/ctypes.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/ctypes.rst Fri Jun 13 16:43:12 2008
@@ -1406,10 +1406,9 @@
to request and change the ctypes private copy of the windows error
code.
-.. versionchanged:: 2.6
-
-The `use_errno` and `use_last_error` parameters were added in Python
-2.6.
+.. versionadded:: 2.6
+ The ``use_last_error`` and ``use_errno`` optional parameters
+ were added.
.. data:: RTLD_GLOBAL
:noindex:
@@ -1572,22 +1571,23 @@
Assign a Python function or another callable to this attribute. The
callable will be called with three or more arguments:
+ .. function:: callable(result, func, arguments)
+ :noindex:
-.. function:: callable(result, func, arguments)
- :noindex:
-
- ``result`` is what the foreign function returns, as specified by the
- :attr:`restype` attribute.
-
- ``func`` is the foreign function object itself, this allows to reuse the same
- callable object to check or post process the results of several functions.
-
- ``arguments`` is a tuple containing the parameters originally passed to the
- function call, this allows to specialize the behavior on the arguments used.
+ ``result`` is what the foreign function returns, as specified
+ by the :attr:`restype` attribute.
- The object that this function returns will be returned from the foreign
- function call, but it can also check the result value and raise an exception
- if the foreign function call failed.
+ ``func`` is the foreign function object itself, this allows
+ to reuse the same callable object to check or post process
+ the results of several functions.
+
+ ``arguments`` is a tuple containing the parameters originally
+ passed to the function call, this allows to specialize the
+ behavior on the arguments used.
+
+ The object that this function returns will be returned from the
+ foreign function call, but it can also check the result value
+ and raise an exception if the foreign function call failed.
.. exception:: ArgumentError()
@@ -1805,12 +1805,22 @@
ctypes type or instance.
-.. function:: byref(obj)
+.. function:: byref(obj[, offset])
- Returns a light-weight pointer to ``obj``, which must be an instance of a ctypes
- type. The returned object can only be used as a foreign function call parameter.
- It behaves similar to ``pointer(obj)``, but the construction is a lot faster.
+ Returns a light-weight pointer to ``obj``, which must be an
+ instance of a ctypes type. ``offset`` defaults to zero, it must be
+ an integer which is added to the internal pointer value.
+ ``byref(obj, offset)`` corresponds to this C code::
+
+ (((char *)&obj) + offset)
+
+ The returned object can only be used as a foreign function call
+ parameter. It behaves similar to ``pointer(obj)``, but the
+ construction is a lot faster.
+
+ .. versionadded:: 2.6
+ The ``offset`` optional argument was added.
.. function:: cast(obj, type)
Modified: python/branches/tlee-ast-optimize/Doc/library/numbers.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/numbers.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/numbers.rst Fri Jun 13 16:43:12 2008
@@ -73,10 +73,10 @@
.. class:: Integral
- Subtypes :class:`Rational` and adds a conversion to :class:`long`, the
- 3-argument form of :func:`pow`, and the bit-string operations: ``<<``,
- ``>>``, ``&``, ``^``, ``|``, ``~``. Provides defaults for :func:`float`,
- :attr:`Rational.numerator`, and :attr:`Rational.denominator`.
+ Subtypes :class:`Rational` and adds a conversion to :class:`int`.
+ Provides defaults for :func:`float`, :attr:`Rational.numerator`, and
+ :attr:`Rational.denominator`, and bit-string operations: ``<<``,
+ ``>>``, ``&``, ``^``, ``|``, ``~``.
Notes for type implementors
Modified: python/branches/tlee-ast-optimize/Doc/library/someos.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/someos.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/someos.rst Fri Jun 13 16:43:12 2008
@@ -18,6 +18,7 @@
threading.rst
dummy_thread.rst
dummy_threading.rst
+ multiprocessing.rst
mmap.rst
readline.rst
rlcompleter.rst
Modified: python/branches/tlee-ast-optimize/Doc/library/stdtypes.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/library/stdtypes.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/library/stdtypes.rst Fri Jun 13 16:43:12 2008
@@ -1583,10 +1583,13 @@
.. versionchanged:: 2.6
Accepts multiple input iterables.
- .. method:: difference(other)
- set - other
+ .. method:: difference(other, ...)
+ set - other - ...
- Return a new set with elements in the set that are not in *other*.
+ Return a new set with elements in the set that are not in the others.
+
+ .. versionchanged:: 2.6
+ Accepts multiple input iterables.
.. method:: symmetric_difference(other)
set ^ other
@@ -1650,10 +1653,13 @@
.. versionchanged:: 2.6
Accepts multiple input iterables.
- .. method:: difference_update(other)
- set -= other
+ .. method:: difference_update(other, ...)
+ set -= other | ...
- Update the set, removing elements found in *other*.
+ Update the set, removing elements found in others.
+
+ .. versionchanged:: 2.6
+ Accepts multiple input iterables.
.. method:: symmetric_difference_update(other)
set ^= other
Modified: python/branches/tlee-ast-optimize/Doc/whatsnew/2.6.rst
==============================================================================
--- python/branches/tlee-ast-optimize/Doc/whatsnew/2.6.rst (original)
+++ python/branches/tlee-ast-optimize/Doc/whatsnew/2.6.rst Fri Jun 13 16:43:12 2008
@@ -521,6 +521,21 @@
.. ======================================================================
+.. _pep-0371:
+
+PEP 371: The ``multiprocessing`` Package
+=====================================================
+
+XXX write this.
+
+.. seealso::
+
+ :pep:`371` - Per-user ``site-packages`` Directory
+ PEP written by Jesse Noller and Richard Oudkerk;
+ implemented by Jesse Noller.
+
+.. ======================================================================
+
.. _pep-3101:
PEP 3101: Advanced String Formatting
Modified: python/branches/tlee-ast-optimize/Include/bytesobject.h
==============================================================================
--- python/branches/tlee-ast-optimize/Include/bytesobject.h (original)
+++ python/branches/tlee-ast-optimize/Include/bytesobject.h Fri Jun 13 16:43:12 2008
@@ -1,200 +1,33 @@
+#define PyBytesObject PyStringObject
+#define PyBytes_Type PyString_Type
-/* String (Bytes) object interface */
-
-#ifndef Py_BYTESOBJECT_H
-#define Py_BYTESOBJECT_H
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-#include <stdarg.h>
-
-/*
-Type PyStringObject represents a character string. An extra zero byte is
-reserved at the end to ensure it is zero-terminated, but a size is
-present so strings with null bytes in them can be represented. This
-is an immutable object type.
-
-There are functions to create new string objects, to test
-an object for string-ness, and to get the
-string value. The latter function returns a null pointer
-if the object is not of the proper type.
-There is a variant that takes an explicit size as well as a
-variant that assumes a zero-terminated string. Note that none of the
-functions should be applied to nil objects.
-*/
-
-/* Caching the hash (ob_shash) saves recalculation of a string's hash value.
- Interning strings (ob_sstate) tries to ensure that only one string
- object with a given value exists, so equality tests can be one pointer
- comparison. This is generally restricted to strings that "look like"
- Python identifiers, although the intern() builtin can be used to force
- interning of any string.
- Together, these sped the interpreter by up to 20%. */
-
-typedef struct {
- PyObject_VAR_HEAD
- long ob_shash;
- int ob_sstate;
- char ob_sval[1];
-
- /* Invariants:
- * ob_sval contains space for 'ob_size+1' elements.
- * ob_sval[ob_size] == 0.
- * ob_shash is the hash of the string or -1 if not computed yet.
- * ob_sstate != 0 iff the string object is in stringobject.c's
- * 'interned' dictionary; in this case the two references
- * from 'interned' to this object are *not counted* in ob_refcnt.
- */
-} PyStringObject;
-
-#define SSTATE_NOT_INTERNED 0
-#define SSTATE_INTERNED_MORTAL 1
-#define SSTATE_INTERNED_IMMORTAL 2
-
-PyAPI_DATA(PyTypeObject) PyBaseString_Type;
-PyAPI_DATA(PyTypeObject) PyString_Type;
-
-#define PyString_Check(op) \
- PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_STRING_SUBCLASS)
-#define PyString_CheckExact(op) (Py_TYPE(op) == &PyString_Type)
-
-PyAPI_FUNC(PyObject *) PyString_FromStringAndSize(const char *, Py_ssize_t);
-PyAPI_FUNC(PyObject *) PyString_FromString(const char *);
-PyAPI_FUNC(PyObject *) PyString_FromFormatV(const char*, va_list)
- Py_GCC_ATTRIBUTE((format(printf, 1, 0)));
-PyAPI_FUNC(PyObject *) PyString_FromFormat(const char*, ...)
- Py_GCC_ATTRIBUTE((format(printf, 1, 2)));
-PyAPI_FUNC(Py_ssize_t) PyString_Size(PyObject *);
-PyAPI_FUNC(char *) PyString_AsString(PyObject *);
-PyAPI_FUNC(PyObject *) PyString_Repr(PyObject *, int);
-PyAPI_FUNC(void) PyString_Concat(PyObject **, PyObject *);
-PyAPI_FUNC(void) PyString_ConcatAndDel(PyObject **, PyObject *);
-PyAPI_FUNC(int) _PyString_Resize(PyObject **, Py_ssize_t);
-PyAPI_FUNC(int) _PyString_Eq(PyObject *, PyObject*);
-PyAPI_FUNC(PyObject *) PyString_Format(PyObject *, PyObject *);
-PyAPI_FUNC(PyObject *) _PyString_FormatLong(PyObject*, int, int,
- int, char**, int*);
-PyAPI_FUNC(PyObject *) PyString_DecodeEscape(const char *, Py_ssize_t,
- const char *, Py_ssize_t,
- const char *);
-
-PyAPI_FUNC(void) PyString_InternInPlace(PyObject **);
-PyAPI_FUNC(void) PyString_InternImmortal(PyObject **);
-PyAPI_FUNC(PyObject *) PyString_InternFromString(const char *);
-PyAPI_FUNC(void) _Py_ReleaseInternedStrings(void);
-
-/* Use only if you know it's a string */
-#define PyString_CHECK_INTERNED(op) (((PyStringObject *)(op))->ob_sstate)
-
-/* Macro, trading safety for speed */
-#define PyString_AS_STRING(op) (((PyStringObject *)(op))->ob_sval)
-#define PyString_GET_SIZE(op) Py_SIZE(op)
-
-/* _PyString_Join(sep, x) is like sep.join(x). sep must be PyStringObject*,
- x must be an iterable object. */
-PyAPI_FUNC(PyObject *) _PyString_Join(PyObject *sep, PyObject *x);
-
-/* --- Generic Codecs ----------------------------------------------------- */
-
-/* Create an object by decoding the encoded string s of the
- given size. */
-
-PyAPI_FUNC(PyObject*) PyString_Decode(
- const char *s, /* encoded string */
- Py_ssize_t size, /* size of buffer */
- const char *encoding, /* encoding */
- const char *errors /* error handling */
- );
-
-/* Encodes a char buffer of the given size and returns a
- Python object. */
-
-PyAPI_FUNC(PyObject*) PyString_Encode(
- const char *s, /* string char buffer */
- Py_ssize_t size, /* number of chars to encode */
- const char *encoding, /* encoding */
- const char *errors /* error handling */
- );
-
-/* Encodes a string object and returns the result as Python
- object. */
-
-PyAPI_FUNC(PyObject*) PyString_AsEncodedObject(
- PyObject *str, /* string object */
- const char *encoding, /* encoding */
- const char *errors /* error handling */
- );
-
-/* Encodes a string object and returns the result as Python string
- object.
-
- If the codec returns an Unicode object, the object is converted
- back to a string using the default encoding.
-
- DEPRECATED - use PyString_AsEncodedObject() instead. */
-
-PyAPI_FUNC(PyObject*) PyString_AsEncodedString(
- PyObject *str, /* string object */
- const char *encoding, /* encoding */
- const char *errors /* error handling */
- );
-
-/* Decodes a string object and returns the result as Python
- object. */
-
-PyAPI_FUNC(PyObject*) PyString_AsDecodedObject(
- PyObject *str, /* string object */
- const char *encoding, /* encoding */
- const char *errors /* error handling */
- );
-
-/* Decodes a string object and returns the result as Python string
- object.
-
- If the codec returns an Unicode object, the object is converted
- back to a string using the default encoding.
-
- DEPRECATED - use PyString_AsDecodedObject() instead. */
-
-PyAPI_FUNC(PyObject*) PyString_AsDecodedString(
- PyObject *str, /* string object */
- const char *encoding, /* encoding */
- const char *errors /* error handling */
- );
-
-/* Provides access to the internal data buffer and size of a string
- object or the default encoded version of an Unicode object. Passing
- NULL as *len parameter will force the string buffer to be
- 0-terminated (passing a string with embedded NULL characters will
- cause an exception). */
-
-PyAPI_FUNC(int) PyString_AsStringAndSize(
- register PyObject *obj, /* string or Unicode object */
- register char **s, /* pointer to buffer variable */
- register Py_ssize_t *len /* pointer to length variable or NULL
- (only possible for 0-terminated
- strings) */
- );
-
-/* Using the current locale, insert the thousands grouping
- into the string pointed to by buffer. For the argument descriptions,
- see Objects/stringlib/localeutil.h */
-
-PyAPI_FUNC(int) _PyString_InsertThousandsGrouping(char *buffer,
- Py_ssize_t len,
- char *plast,
- Py_ssize_t buf_size,
- Py_ssize_t *count,
- int append_zero_char);
-
-/* Format the object based on the format_spec, as defined in PEP 3101
- (Advanced String Formatting). */
-PyAPI_FUNC(PyObject *) _PyBytes_FormatAdvanced(PyObject *obj,
- char *format_spec,
- Py_ssize_t format_spec_len);
-
-#ifdef __cplusplus
-}
-#endif
-#endif /* !Py_BYTESOBJECT_H */
+#define PyBytes_Check PyString_Check
+#define PyBytes_CheckExact PyString_CheckExact
+#define PyBytes_CHECK_INTERNED PyString_CHECK_INTERNED
+#define PyBytes_AS_STRING PyString_AS_STRING
+#define PyBytes_GET_SIZE PyString_GET_SIZE
+#define Py_TPFLAGS_BYTES_SUBCLASS Py_TPFLAGS_STRING_SUBCLASS
+
+#define PyBytes_FromStringAndSize PyString_FromStringAndSize
+#define PyBytes_FromString PyString_FromString
+#define PyBytes_FromFormatV PyString_FromFormatV
+#define PyBytes_FromFormat PyString_FromFormat
+#define PyBytes_Size PyString_Size
+#define PyBytes_AsString PyString_AsString
+#define PyBytes_Repr PyString_Repr
+#define PyBytes_Concat PyString_Concat
+#define PyBytes_ConcatAndDel PyString_ConcatAndDel
+#define _PyBytes_Resize _PyString_Resize
+#define _PyBytes_Eq _PyString_Eq
+#define PyBytes_Format PyString_Format
+#define _PyBytes_FormatLong _PyString_FormatLong
+#define PyBytes_DecodeEscape PyString_DecodeEscape
+#define _PyBytes_Join _PyString_Join
+#define PyBytes_Decode PyString_Decode
+#define PyBytes_Encode PyString_Encode
+#define PyBytes_AsEncodedObject PyString_AsEncodedObject
+#define PyBytes_AsEncodedString PyString_AsEncodedString
+#define PyBytes_AsDecodedObject PyString_AsDecodedObject
+#define PyBytes_AsDecodedString PyString_AsDecodedString
+#define PyBytes_AsStringAndSize PyString_AsStringAndSize
+#define _PyBytes_InsertThousandsGrouping _PyString_InsertThousandsGrouping
Modified: python/branches/tlee-ast-optimize/Include/pymem.h
==============================================================================
--- python/branches/tlee-ast-optimize/Include/pymem.h (original)
+++ python/branches/tlee-ast-optimize/Include/pymem.h Fri Jun 13 16:43:12 2008
@@ -85,14 +85,18 @@
*/
#define PyMem_New(type, n) \
- ( (type *) PyMem_Malloc((n) * sizeof(type)) )
+ ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
+ ( (type *) PyMem_Malloc((n) * sizeof(type)) ) )
#define PyMem_NEW(type, n) \
- ( (type *) PyMem_MALLOC((n) * sizeof(type)) )
+ ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
+ ( (type *) PyMem_MALLOC((n) * sizeof(type)) ) )
#define PyMem_Resize(p, type, n) \
- ( (p) = (type *) PyMem_Realloc((p), (n) * sizeof(type)) )
+ ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
+ ( (p) = (type *) PyMem_Realloc((p), (n) * sizeof(type)) ) )
#define PyMem_RESIZE(p, type, n) \
- ( (p) = (type *) PyMem_REALLOC((p), (n) * sizeof(type)) )
+ ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
+ ( (p) = (type *) PyMem_REALLOC((p), (n) * sizeof(type)) ) )
/* PyMem{Del,DEL} are left over from ancient days, and shouldn't be used
* anymore. They're just confusing aliases for PyMem_{Free,FREE} now.
Modified: python/branches/tlee-ast-optimize/Include/pyport.h
==============================================================================
--- python/branches/tlee-ast-optimize/Include/pyport.h (original)
+++ python/branches/tlee-ast-optimize/Include/pyport.h Fri Jun 13 16:43:12 2008
@@ -117,6 +117,17 @@
# error "Python needs a typedef for Py_ssize_t in pyport.h."
#endif
+/* Largest possible value of size_t.
+ SIZE_MAX is part of C99, so it might be defined on some
+ platforms. If it is not defined, (size_t)-1 is a portable
+ definition for C89, due to the way signed->unsigned
+ conversion is defined. */
+#ifdef SIZE_MAX
+#define PY_SIZE_MAX SIZE_MAX
+#else
+#define PY_SIZE_MAX ((size_t)-1)
+#endif
+
/* Largest positive value of type Py_ssize_t. */
#define PY_SSIZE_T_MAX ((Py_ssize_t)(((size_t)-1)>>1))
/* Smallest negative value of type Py_ssize_t. */
Modified: python/branches/tlee-ast-optimize/Include/stringobject.h
==============================================================================
--- python/branches/tlee-ast-optimize/Include/stringobject.h (original)
+++ python/branches/tlee-ast-optimize/Include/stringobject.h Fri Jun 13 16:43:12 2008
@@ -1,33 +1,200 @@
-#define PyBytesObject PyStringObject
-#define PyBytes_Type PyString_Type
-#define PyBytes_Check PyString_Check
-#define PyBytes_CheckExact PyString_CheckExact
-#define PyBytes_CHECK_INTERNED PyString_CHECK_INTERNED
-#define PyBytes_AS_STRING PyString_AS_STRING
-#define PyBytes_GET_SIZE PyString_GET_SIZE
-#define Py_TPFLAGS_BYTES_SUBCLASS Py_TPFLAGS_STRING_SUBCLASS
-
-#define PyBytes_FromStringAndSize PyString_FromStringAndSize
-#define PyBytes_FromString PyString_FromString
-#define PyBytes_FromFormatV PyString_FromFormatV
-#define PyBytes_FromFormat PyString_FromFormat
-#define PyBytes_Size PyString_Size
-#define PyBytes_AsString PyString_AsString
-#define PyBytes_Repr PyString_Repr
-#define PyBytes_Concat PyString_Concat
-#define PyBytes_ConcatAndDel PyString_ConcatAndDel
-#define _PyBytes_Resize _PyString_Resize
-#define _PyBytes_Eq _PyString_Eq
-#define PyBytes_Format PyString_Format
-#define _PyBytes_FormatLong _PyString_FormatLong
-#define PyBytes_DecodeEscape PyString_DecodeEscape
-#define _PyBytes_Join _PyString_Join
-#define PyBytes_Decode PyString_Decode
-#define PyBytes_Encode PyString_Encode
-#define PyBytes_AsEncodedObject PyString_AsEncodedObject
-#define PyBytes_AsEncodedString PyString_AsEncodedString
-#define PyBytes_AsDecodedObject PyString_AsDecodedObject
-#define PyBytes_AsDecodedString PyString_AsDecodedString
-#define PyBytes_AsStringAndSize PyString_AsStringAndSize
-#define _PyBytes_InsertThousandsGrouping _PyString_InsertThousandsGrouping
+/* String (str/bytes) object interface */
+
+#ifndef Py_STRINGOBJECT_H
+#define Py_STRINGOBJECT_H
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdarg.h>
+
+/*
+Type PyStringObject represents a character string. An extra zero byte is
+reserved at the end to ensure it is zero-terminated, but a size is
+present so strings with null bytes in them can be represented. This
+is an immutable object type.
+
+There are functions to create new string objects, to test
+an object for string-ness, and to get the
+string value. The latter function returns a null pointer
+if the object is not of the proper type.
+There is a variant that takes an explicit size as well as a
+variant that assumes a zero-terminated string. Note that none of the
+functions should be applied to nil objects.
+*/
+
+/* Caching the hash (ob_shash) saves recalculation of a string's hash value.
+ Interning strings (ob_sstate) tries to ensure that only one string
+ object with a given value exists, so equality tests can be one pointer
+ comparison. This is generally restricted to strings that "look like"
+ Python identifiers, although the intern() builtin can be used to force
+ interning of any string.
+ Together, these sped the interpreter by up to 20%. */
+
+typedef struct {
+ PyObject_VAR_HEAD
+ long ob_shash;
+ int ob_sstate;
+ char ob_sval[1];
+
+ /* Invariants:
+ * ob_sval contains space for 'ob_size+1' elements.
+ * ob_sval[ob_size] == 0.
+ * ob_shash is the hash of the string or -1 if not computed yet.
+ * ob_sstate != 0 iff the string object is in stringobject.c's
+ * 'interned' dictionary; in this case the two references
+ * from 'interned' to this object are *not counted* in ob_refcnt.
+ */
+} PyStringObject;
+
+#define SSTATE_NOT_INTERNED 0
+#define SSTATE_INTERNED_MORTAL 1
+#define SSTATE_INTERNED_IMMORTAL 2
+
+PyAPI_DATA(PyTypeObject) PyBaseString_Type;
+PyAPI_DATA(PyTypeObject) PyString_Type;
+
+#define PyString_Check(op) \
+ PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_STRING_SUBCLASS)
+#define PyString_CheckExact(op) (Py_TYPE(op) == &PyString_Type)
+
+PyAPI_FUNC(PyObject *) PyString_FromStringAndSize(const char *, Py_ssize_t);
+PyAPI_FUNC(PyObject *) PyString_FromString(const char *);
+PyAPI_FUNC(PyObject *) PyString_FromFormatV(const char*, va_list)
+ Py_GCC_ATTRIBUTE((format(printf, 1, 0)));
+PyAPI_FUNC(PyObject *) PyString_FromFormat(const char*, ...)
+ Py_GCC_ATTRIBUTE((format(printf, 1, 2)));
+PyAPI_FUNC(Py_ssize_t) PyString_Size(PyObject *);
+PyAPI_FUNC(char *) PyString_AsString(PyObject *);
+PyAPI_FUNC(PyObject *) PyString_Repr(PyObject *, int);
+PyAPI_FUNC(void) PyString_Concat(PyObject **, PyObject *);
+PyAPI_FUNC(void) PyString_ConcatAndDel(PyObject **, PyObject *);
+PyAPI_FUNC(int) _PyString_Resize(PyObject **, Py_ssize_t);
+PyAPI_FUNC(int) _PyString_Eq(PyObject *, PyObject*);
+PyAPI_FUNC(PyObject *) PyString_Format(PyObject *, PyObject *);
+PyAPI_FUNC(PyObject *) _PyString_FormatLong(PyObject*, int, int,
+ int, char**, int*);
+PyAPI_FUNC(PyObject *) PyString_DecodeEscape(const char *, Py_ssize_t,
+ const char *, Py_ssize_t,
+ const char *);
+
+PyAPI_FUNC(void) PyString_InternInPlace(PyObject **);
+PyAPI_FUNC(void) PyString_InternImmortal(PyObject **);
+PyAPI_FUNC(PyObject *) PyString_InternFromString(const char *);
+PyAPI_FUNC(void) _Py_ReleaseInternedStrings(void);
+
+/* Use only if you know it's a string */
+#define PyString_CHECK_INTERNED(op) (((PyStringObject *)(op))->ob_sstate)
+
+/* Macro, trading safety for speed */
+#define PyString_AS_STRING(op) (((PyStringObject *)(op))->ob_sval)
+#define PyString_GET_SIZE(op) Py_SIZE(op)
+
+/* _PyString_Join(sep, x) is like sep.join(x). sep must be PyStringObject*,
+ x must be an iterable object. */
+PyAPI_FUNC(PyObject *) _PyString_Join(PyObject *sep, PyObject *x);
+
+/* --- Generic Codecs ----------------------------------------------------- */
+
+/* Create an object by decoding the encoded string s of the
+ given size. */
+
+PyAPI_FUNC(PyObject*) PyString_Decode(
+ const char *s, /* encoded string */
+ Py_ssize_t size, /* size of buffer */
+ const char *encoding, /* encoding */
+ const char *errors /* error handling */
+ );
+
+/* Encodes a char buffer of the given size and returns a
+ Python object. */
+
+PyAPI_FUNC(PyObject*) PyString_Encode(
+ const char *s, /* string char buffer */
+ Py_ssize_t size, /* number of chars to encode */
+ const char *encoding, /* encoding */
+ const char *errors /* error handling */
+ );
+
+/* Encodes a string object and returns the result as Python
+ object. */
+
+PyAPI_FUNC(PyObject*) PyString_AsEncodedObject(
+ PyObject *str, /* string object */
+ const char *encoding, /* encoding */
+ const char *errors /* error handling */
+ );
+
+/* Encodes a string object and returns the result as Python string
+ object.
+
+ If the codec returns an Unicode object, the object is converted
+ back to a string using the default encoding.
+
+ DEPRECATED - use PyString_AsEncodedObject() instead. */
+
+PyAPI_FUNC(PyObject*) PyString_AsEncodedString(
+ PyObject *str, /* string object */
+ const char *encoding, /* encoding */
+ const char *errors /* error handling */
+ );
+
+/* Decodes a string object and returns the result as Python
+ object. */
+
+PyAPI_FUNC(PyObject*) PyString_AsDecodedObject(
+ PyObject *str, /* string object */
+ const char *encoding, /* encoding */
+ const char *errors /* error handling */
+ );
+
+/* Decodes a string object and returns the result as Python string
+ object.
+
+ If the codec returns an Unicode object, the object is converted
+ back to a string using the default encoding.
+
+ DEPRECATED - use PyString_AsDecodedObject() instead. */
+
+PyAPI_FUNC(PyObject*) PyString_AsDecodedString(
+ PyObject *str, /* string object */
+ const char *encoding, /* encoding */
+ const char *errors /* error handling */
+ );
+
+/* Provides access to the internal data buffer and size of a string
+ object or the default encoded version of an Unicode object. Passing
+ NULL as *len parameter will force the string buffer to be
+ 0-terminated (passing a string with embedded NULL characters will
+ cause an exception). */
+
+PyAPI_FUNC(int) PyString_AsStringAndSize(
+ register PyObject *obj, /* string or Unicode object */
+ register char **s, /* pointer to buffer variable */
+ register Py_ssize_t *len /* pointer to length variable or NULL
+ (only possible for 0-terminated
+ strings) */
+ );
+
+/* Using the current locale, insert the thousands grouping
+ into the string pointed to by buffer. For the argument descriptions,
+ see Objects/stringlib/localeutil.h */
+
+PyAPI_FUNC(int) _PyString_InsertThousandsGrouping(char *buffer,
+ Py_ssize_t len,
+ char *plast,
+ Py_ssize_t buf_size,
+ Py_ssize_t *count,
+ int append_zero_char);
+
+/* Format the object based on the format_spec, as defined in PEP 3101
+ (Advanced String Formatting). */
+PyAPI_FUNC(PyObject *) _PyBytes_FormatAdvanced(PyObject *obj,
+ char *format_spec,
+ Py_ssize_t format_spec_len);
+
+#ifdef __cplusplus
+}
+#endif
+#endif /* !Py_STRINGOBJECT_H */
Modified: python/branches/tlee-ast-optimize/Lib/asyncore.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/asyncore.py (original)
+++ python/branches/tlee-ast-optimize/Lib/asyncore.py Fri Jun 13 16:43:12 2008
@@ -228,7 +228,7 @@
# passed be connected.
try:
self.addr = sock.getpeername()
- except socket.error:
+ except socket.error, err:
if err[0] == ENOTCONN:
# To handle the case where we got an unconnected
# socket.
@@ -424,7 +424,7 @@
#check for errors
err = self.socket.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
if err != 0:
- raise socket.error(err, strerror(err))
+ raise socket.error(err, _strerror(err))
self.handle_connect_event()
self.handle_write()
Modified: python/branches/tlee-ast-optimize/Lib/numbers.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/numbers.py (original)
+++ python/branches/tlee-ast-optimize/Lib/numbers.py Fri Jun 13 16:43:12 2008
@@ -283,87 +283,54 @@
class Integral(Rational):
- """Integral adds a conversion to long and the bit-string operations."""
+ """Integral adds a conversion to int and the bit-string operations."""
@abstractmethod
- def __long__(self):
- """long(self)"""
+ def __int__(self):
+ """int(self)"""
raise NotImplementedError
def __index__(self):
"""index(self)"""
- return long(self)
+ return int(self)
- @abstractmethod
- def __pow__(self, exponent, modulus=None):
- """self ** exponent % modulus, but maybe faster.
-
- Accept the modulus argument if you want to support the
- 3-argument version of pow(). Raise a TypeError if exponent < 0
- or any argument isn't Integral. Otherwise, just implement the
- 2-argument version described in Complex.
- """
- raise NotImplementedError
-
- @abstractmethod
def __lshift__(self, other):
- """self << other"""
- raise NotImplementedError
+ return int(self) << int(other)
- @abstractmethod
def __rlshift__(self, other):
- """other << self"""
- raise NotImplementedError
+ return int(other) << int(self)
- @abstractmethod
def __rshift__(self, other):
- """self >> other"""
- raise NotImplementedError
+ return int(self) >> int(other)
- @abstractmethod
def __rrshift__(self, other):
- """other >> self"""
- raise NotImplementedError
+ return int(other) >> int(self)
- @abstractmethod
def __and__(self, other):
- """self & other"""
- raise NotImplementedError
+ return int(self) & int(other)
- @abstractmethod
def __rand__(self, other):
- """other & self"""
- raise NotImplementedError
+ return int(other) & int(self)
- @abstractmethod
def __xor__(self, other):
- """self ^ other"""
- raise NotImplementedError
+ return int(self) ^ int(other)
- @abstractmethod
def __rxor__(self, other):
- """other ^ self"""
- raise NotImplementedError
+ return int(other) ^ int(self)
- @abstractmethod
def __or__(self, other):
- """self | other"""
- raise NotImplementedError
+ return int(self) | int(other)
- @abstractmethod
def __ror__(self, other):
- """other | self"""
- raise NotImplementedError
+ return int(other) | int(self)
- @abstractmethod
def __invert__(self):
- """~self"""
- raise NotImplementedError
+ return ~int(self)
# Concrete implementations of Rational and Real abstract methods.
def __float__(self):
- """float(self) == float(long(self))"""
- return float(long(self))
+ """float(self) == float(int(self))"""
+ return float(int(self))
@property
def numerator(self):
Modified: python/branches/tlee-ast-optimize/Lib/test/test_array.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_array.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_array.py Fri Jun 13 16:43:12 2008
@@ -1009,6 +1009,23 @@
class DoubleTest(FPTest):
typecode = 'd'
minitemsize = 8
+
+ def test_alloc_overflow(self):
+ a = array.array('d', [-1]*65536)
+ try:
+ a *= 65536
+ except MemoryError:
+ pass
+ else:
+ self.fail("a *= 2**16 didn't raise MemoryError")
+ b = array.array('d', [ 2.71828183, 3.14159265, -1])
+ try:
+ b * 1431655766
+ except MemoryError:
+ pass
+ else:
+ self.fail("a * 1431655766 didn't raise MemoryError")
+
tests.append(DoubleTest)
def test_main(verbose=None):
Modified: python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_asyncore.py Fri Jun 13 16:43:12 2008
@@ -384,8 +384,8 @@
fd = os.open(TESTFN, os.O_RDONLY)
w = asyncore.file_wrapper(fd)
- self.assertEqual(w.fd, fd)
- self.assertEqual(w.fileno(), fd)
+ self.assertNotEqual(w.fd, fd)
+ self.assertNotEqual(w.fileno(), fd)
self.assertEqual(w.recv(13), "It's not dead")
self.assertEqual(w.read(6), ", it's")
w.close()
Modified: python/branches/tlee-ast-optimize/Lib/test/test_heapq.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_heapq.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_heapq.py Fri Jun 13 16:43:12 2008
@@ -196,6 +196,27 @@
class TestHeapC(TestHeap):
module = c_heapq
+ def test_comparison_operator(self):
+ # Issue 3501: Make sure heapq works with both __lt__ and __le__
+ def hsort(data, comp):
+ data = map(comp, data)
+ self.module.heapify(data)
+ return [self.module.heappop(data).x for i in range(len(data))]
+ class LT:
+ def __init__(self, x):
+ self.x = x
+ def __lt__(self, other):
+ return self.x > other.x
+ class LE:
+ def __init__(self, x):
+ self.x = x
+ def __lt__(self, other):
+ return self.x >= other.x
+ data = [random.random() for i in range(100)]
+ target = sorted(data, reverse=True)
+ self.assertEqual(hsort(data, LT), target)
+ self.assertEqual(hsort(data, LE), target)
+
#==============================================================================
Modified: python/branches/tlee-ast-optimize/Lib/test/test_platform.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_platform.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_platform.py Fri Jun 13 16:43:12 2008
@@ -72,7 +72,7 @@
else:
have_toolbox_glue = True
- if have_toolbox_glue and os.uname()[0] == 'Darwin':
+ if have_toolbox_glue and platform.uname()[0] == 'Darwin':
# We're on a MacOSX system, check that
# the right version information is returned
fd = os.popen('sw_vers', 'r')
Modified: python/branches/tlee-ast-optimize/Lib/test/test_pydoc.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_pydoc.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_pydoc.py Fri Jun 13 16:43:12 2008
@@ -214,7 +214,12 @@
def test_html_doc(self):
result, doc_loc = get_pydoc_html(pydoc_mod)
mod_file = inspect.getabsfile(pydoc_mod)
- expected_html = expected_html_pattern % (mod_file, mod_file, doc_loc)
+ if sys.platform == 'win32':
+ import nturl2path
+ mod_url = nturl2path.pathname2url(mod_file)
+ else:
+ mod_url = mod_file
+ expected_html = expected_html_pattern % (mod_url, mod_file, doc_loc)
if result != expected_html:
print_diffs(expected_html, result)
self.fail("outputs are not equal, see diff above")
Modified: python/branches/tlee-ast-optimize/Lib/test/test_set.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_set.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_set.py Fri Jun 13 16:43:12 2008
@@ -104,6 +104,12 @@
self.assertEqual(self.thetype('abcba').intersection(C('ccb')), set('bc'))
self.assertEqual(self.thetype('abcba').intersection(C('ef')), set(''))
self.assertEqual(self.thetype('abcba').intersection(C('cbcf'), C('bag')), set('b'))
+ s = self.thetype('abcba')
+ z = s.intersection()
+ if self.thetype == frozenset():
+ self.assertEqual(id(s), id(z))
+ else:
+ self.assertNotEqual(id(s), id(z))
def test_isdisjoint(self):
def f(s1, s2):
@@ -143,6 +149,8 @@
self.assertEqual(self.thetype('abcba').difference(C('efgfe')), set('abc'))
self.assertEqual(self.thetype('abcba').difference(C('ccb')), set('a'))
self.assertEqual(self.thetype('abcba').difference(C('ef')), set('abc'))
+ self.assertEqual(self.thetype('abcba').difference(), set('abc'))
+ self.assertEqual(self.thetype('abcba').difference(C('a'), C('b')), set('c'))
def test_sub(self):
i = self.s.difference(self.otherword)
@@ -461,6 +469,18 @@
self.assertEqual(s.difference_update(C(p)), None)
self.assertEqual(s, set(q))
+ s = self.thetype('abcdefghih')
+ s.difference_update()
+ self.assertEqual(s, self.thetype('abcdefghih'))
+
+ s = self.thetype('abcdefghih')
+ s.difference_update(C('aba'))
+ self.assertEqual(s, self.thetype('cdefghih'))
+
+ s = self.thetype('abcdefghih')
+ s.difference_update(C('cdc'), C('aba'))
+ self.assertEqual(s, self.thetype('efghih'))
+
def test_isub(self):
self.s -= set(self.otherword)
for c in (self.word + self.otherword):
Modified: python/branches/tlee-ast-optimize/Lib/test/test_struct.py
==============================================================================
--- python/branches/tlee-ast-optimize/Lib/test/test_struct.py (original)
+++ python/branches/tlee-ast-optimize/Lib/test/test_struct.py Fri Jun 13 16:43:12 2008
@@ -1,14 +1,15 @@
-from test.test_support import TestFailed, verbose, verify, vereq
-import test.test_support
-import struct
import array
+import unittest
+import struct
import warnings
+from functools import wraps
+from test.test_support import TestFailed, verbose, run_unittest, catch_warning
+
import sys
ISBIGENDIAN = sys.byteorder == "big"
+IS32BIT = sys.maxint == 0x7fffffff
del sys
-verify((struct.pack('=i', 1)[0] == chr(0)) == ISBIGENDIAN,
- "bigendian determination appears wrong")
try:
import _struct
@@ -30,39 +31,21 @@
else:
return string_reverse(value)
-def simple_err(func, *args):
- try:
- func(*args)
- except struct.error:
- pass
- else:
- raise TestFailed, "%s%s did not raise struct.error" % (
- func.__name__, args)
-
-def any_err(func, *args):
- try:
- func(*args)
- except (struct.error, TypeError):
- pass
- else:
- raise TestFailed, "%s%s did not raise error" % (
- func.__name__, args)
-
def with_warning_restore(func):
- def _with_warning_restore(*args, **kw):
- with test.test_support.catch_warning():
+ @wraps(func)
+ def decorator(*args, **kw):
+ with catch_warning():
# Grrr, we need this function to warn every time. Without removing
# the warningregistry, running test_tarfile then test_struct would fail
# on 64-bit platforms.
globals = func.func_globals
if '__warningregistry__' in globals:
del globals['__warningregistry__']
- warnings.filterwarnings("error", r"""^struct.*""", DeprecationWarning)
- warnings.filterwarnings("error", r""".*format requires.*""",
- DeprecationWarning)
+ warnings.filterwarnings("error", category=DeprecationWarning)
return func(*args, **kw)
- return _with_warning_restore
+ return decorator
+ at with_warning_restore
def deprecated_err(func, *args):
try:
func(*args)
@@ -75,602 +58,527 @@
else:
raise TestFailed, "%s%s did not raise error" % (
func.__name__, args)
-deprecated_err = with_warning_restore(deprecated_err)
-
-simple_err(struct.calcsize, 'Z')
-sz = struct.calcsize('i')
-if sz * 3 != struct.calcsize('iii'):
- raise TestFailed, 'inconsistent sizes'
-
-fmt = 'cbxxxxxxhhhhiillffd?'
-fmt3 = '3c3b18x12h6i6l6f3d3?'
-sz = struct.calcsize(fmt)
-sz3 = struct.calcsize(fmt3)
-if sz * 3 != sz3:
- raise TestFailed, 'inconsistent sizes (3*%r -> 3*%d = %d, %r -> %d)' % (
- fmt, sz, 3*sz, fmt3, sz3)
-
-simple_err(struct.pack, 'iii', 3)
-simple_err(struct.pack, 'i', 3, 3, 3)
-simple_err(struct.pack, 'i', 'foo')
-simple_err(struct.pack, 'P', 'foo')
-simple_err(struct.unpack, 'd', 'flap')
-s = struct.pack('ii', 1, 2)
-simple_err(struct.unpack, 'iii', s)
-simple_err(struct.unpack, 'i', s)
-
-c = 'a'
-b = 1
-h = 255
-i = 65535
-l = 65536
-f = 3.1415
-d = 3.1415
-t = True
-
-for prefix in ('', '@', '<', '>', '=', '!'):
- for format in ('xcbhilfd?', 'xcBHILfd?'):
- format = prefix + format
- if verbose:
- print "trying:", format
- s = struct.pack(format, c, b, h, i, l, f, d, t)
- cp, bp, hp, ip, lp, fp, dp, tp = struct.unpack(format, s)
- if (cp != c or bp != b or hp != h or ip != i or lp != l or
- int(100 * fp) != int(100 * f) or int(100 * dp) != int(100 * d) or
- tp != t):
- # ^^^ calculate only to two decimal places
- raise TestFailed, "unpack/pack not transitive (%s, %s)" % (
- str(format), str((cp, bp, hp, ip, lp, fp, dp, tp)))
-
-# Test some of the new features in detail
-
-# (format, argument, big-endian result, little-endian result, asymmetric)
-tests = [
- ('c', 'a', 'a', 'a', 0),
- ('xc', 'a', '\0a', '\0a', 0),
- ('cx', 'a', 'a\0', 'a\0', 0),
- ('s', 'a', 'a', 'a', 0),
- ('0s', 'helloworld', '', '', 1),
- ('1s', 'helloworld', 'h', 'h', 1),
- ('9s', 'helloworld', 'helloworl', 'helloworl', 1),
- ('10s', 'helloworld', 'helloworld', 'helloworld', 0),
- ('11s', 'helloworld', 'helloworld\0', 'helloworld\0', 1),
- ('20s', 'helloworld', 'helloworld'+10*'\0', 'helloworld'+10*'\0', 1),
- ('b', 7, '\7', '\7', 0),
- ('b', -7, '\371', '\371', 0),
- ('B', 7, '\7', '\7', 0),
- ('B', 249, '\371', '\371', 0),
- ('h', 700, '\002\274', '\274\002', 0),
- ('h', -700, '\375D', 'D\375', 0),
- ('H', 700, '\002\274', '\274\002', 0),
- ('H', 0x10000-700, '\375D', 'D\375', 0),
- ('i', 70000000, '\004,\035\200', '\200\035,\004', 0),
- ('i', -70000000, '\373\323\342\200', '\200\342\323\373', 0),
- ('I', 70000000L, '\004,\035\200', '\200\035,\004', 0),
- ('I', 0x100000000L-70000000, '\373\323\342\200', '\200\342\323\373', 0),
- ('l', 70000000, '\004,\035\200', '\200\035,\004', 0),
- ('l', -70000000, '\373\323\342\200', '\200\342\323\373', 0),
- ('L', 70000000L, '\004,\035\200', '\200\035,\004', 0),
- ('L', 0x100000000L-70000000, '\373\323\342\200', '\200\342\323\373', 0),
- ('f', 2.0, '@\000\000\000', '\000\000\000@', 0),
- ('d', 2.0, '@\000\000\000\000\000\000\000',
- '\000\000\000\000\000\000\000@', 0),
- ('f', -2.0, '\300\000\000\000', '\000\000\000\300', 0),
- ('d', -2.0, '\300\000\000\000\000\000\000\000',
- '\000\000\000\000\000\000\000\300', 0),
- ('?', 0, '\0', '\0', 0),
- ('?', 3, '\1', '\1', 1),
- ('?', True, '\1', '\1', 0),
- ('?', [], '\0', '\0', 1),
- ('?', (1,), '\1', '\1', 1),
-]
-
-for fmt, arg, big, lil, asy in tests:
- if verbose:
- print "%r %r %r %r" % (fmt, arg, big, lil)
- for (xfmt, exp) in [('>'+fmt, big), ('!'+fmt, big), ('<'+fmt, lil),
- ('='+fmt, ISBIGENDIAN and big or lil)]:
- res = struct.pack(xfmt, arg)
- if res != exp:
- raise TestFailed, "pack(%r, %r) -> %r # expected %r" % (
- fmt, arg, res, exp)
- n = struct.calcsize(xfmt)
- if n != len(res):
- raise TestFailed, "calcsize(%r) -> %d # expected %d" % (
- xfmt, n, len(res))
- rev = struct.unpack(xfmt, res)[0]
- if rev != arg and not asy:
- raise TestFailed, "unpack(%r, %r) -> (%r,) # expected (%r,)" % (
- fmt, res, rev, arg)
-
-###########################################################################
-# Simple native q/Q tests.
-
-has_native_qQ = 1
-try:
- struct.pack("q", 5)
-except struct.error:
- has_native_qQ = 0
-
-if verbose:
- print "Platform has native q/Q?", has_native_qQ and "Yes." or "No."
-
-any_err(struct.pack, "Q", -1) # can't pack -1 as unsigned regardless
-simple_err(struct.pack, "q", "a") # can't pack string as 'q' regardless
-simple_err(struct.pack, "Q", "a") # ditto, but 'Q'
-
-def test_native_qQ():
- bytes = struct.calcsize('q')
- # The expected values here are in big-endian format, primarily because
- # I'm on a little-endian machine and so this is the clearest way (for
- # me) to force the code to get exercised.
- for format, input, expected in (
- ('q', -1, '\xff' * bytes),
- ('q', 0, '\x00' * bytes),
- ('Q', 0, '\x00' * bytes),
- ('q', 1L, '\x00' * (bytes-1) + '\x01'),
- ('Q', (1L << (8*bytes))-1, '\xff' * bytes),
- ('q', (1L << (8*bytes-1))-1, '\x7f' + '\xff' * (bytes - 1))):
- got = struct.pack(format, input)
- native_expected = bigendian_to_native(expected)
- verify(got == native_expected,
- "%r-pack of %r gave %r, not %r" %
- (format, input, got, native_expected))
- retrieved = struct.unpack(format, got)[0]
- verify(retrieved == input,
- "%r-unpack of %r gave %r, not %r" %
- (format, got, retrieved, input))
-
-if has_native_qQ:
- test_native_qQ()
-
-###########################################################################
-# Standard integer tests (bBhHiIlLqQ).
-
-import binascii
-
-class IntTester:
-
- # XXX Most std integer modes fail to test for out-of-range.
- # The "i" and "l" codes appear to range-check OK on 32-bit boxes, but
- # fail to check correctly on some 64-bit ones (Tru64 Unix + Compaq C
- # reported by Mark Favas).
- BUGGY_RANGE_CHECK = "bBhHiIlL"
-
- def __init__(self, formatpair, bytesize):
- assert len(formatpair) == 2
- self.formatpair = formatpair
- for direction in "<>!=":
- for code in formatpair:
- format = direction + code
- verify(struct.calcsize(format) == bytesize)
- self.bytesize = bytesize
- self.bitsize = bytesize * 8
- self.signed_code, self.unsigned_code = formatpair
- self.unsigned_min = 0
- self.unsigned_max = 2L**self.bitsize - 1
- self.signed_min = -(2L**(self.bitsize-1))
- self.signed_max = 2L**(self.bitsize-1) - 1
-
- def test_one(self, x, pack=struct.pack,
- unpack=struct.unpack,
- unhexlify=binascii.unhexlify):
- if verbose:
- print "trying std", self.formatpair, "on", x, "==", hex(x)
-
- # Try signed.
- code = self.signed_code
- if self.signed_min <= x <= self.signed_max:
- # Try big-endian.
- expected = long(x)
- if x < 0:
- expected += 1L << self.bitsize
- assert expected > 0
- expected = hex(expected)[2:-1] # chop "0x" and trailing 'L'
- if len(expected) & 1:
- expected = "0" + expected
- expected = unhexlify(expected)
- expected = "\x00" * (self.bytesize - len(expected)) + expected
-
- # Pack work?
- format = ">" + code
- got = pack(format, x)
- verify(got == expected,
- "'%s'-pack of %r gave %r, not %r" %
- (format, x, got, expected))
-
- # Unpack work?
- retrieved = unpack(format, got)[0]
- verify(x == retrieved,
- "'%s'-unpack of %r gave %r, not %r" %
- (format, got, retrieved, x))
-
- # Adding any byte should cause a "too big" error.
- any_err(unpack, format, '\x01' + got)
-
- # Try little-endian.
- format = "<" + code
- expected = string_reverse(expected)
-
- # Pack work?
- got = pack(format, x)
- verify(got == expected,
- "'%s'-pack of %r gave %r, not %r" %
- (format, x, got, expected))
-
- # Unpack work?
- retrieved = unpack(format, got)[0]
- verify(x == retrieved,
- "'%s'-unpack of %r gave %r, not %r" %
- (format, got, retrieved, x))
-
- # Adding any byte should cause a "too big" error.
- any_err(unpack, format, '\x01' + got)
+class StructTest(unittest.TestCase):
+ @with_warning_restore
+ def check_float_coerce(self, format, number):
+ # SF bug 1530559. struct.pack raises TypeError where it used to convert.
+ if PY_STRUCT_FLOAT_COERCE == 2:
+ # Test for pre-2.5 struct module
+ packed = struct.pack(format, number)
+ floored = struct.unpack(format, packed)[0]
+ self.assertEqual(floored, int(number),
+ "did not correcly coerce float to int")
+ return
+ try:
+ struct.pack(format, number)
+ except (struct.error, TypeError):
+ if PY_STRUCT_FLOAT_COERCE:
+ self.fail("expected DeprecationWarning for float coerce")
+ except DeprecationWarning:
+ if not PY_STRUCT_FLOAT_COERCE:
+ self.fail("expected to raise struct.error for float coerce")
else:
- # x is out of range -- verify pack realizes that.
- if not PY_STRUCT_RANGE_CHECKING and code in self.BUGGY_RANGE_CHECK:
- if verbose:
- print "Skipping buggy range check for code", code
- else:
- deprecated_err(pack, ">" + code, x)
- deprecated_err(pack, "<" + code, x)
-
- # Much the same for unsigned.
- code = self.unsigned_code
- if self.unsigned_min <= x <= self.unsigned_max:
- # Try big-endian.
- format = ">" + code
- expected = long(x)
- expected = hex(expected)[2:-1] # chop "0x" and trailing 'L'
- if len(expected) & 1:
- expected = "0" + expected
- expected = unhexlify(expected)
- expected = "\x00" * (self.bytesize - len(expected)) + expected
-
- # Pack work?
- got = pack(format, x)
- verify(got == expected,
- "'%s'-pack of %r gave %r, not %r" %
- (format, x, got, expected))
-
- # Unpack work?
- retrieved = unpack(format, got)[0]
- verify(x == retrieved,
- "'%s'-unpack of %r gave %r, not %r" %
- (format, got, retrieved, x))
-
- # Adding any byte should cause a "too big" error.
- any_err(unpack, format, '\x01' + got)
-
- # Try little-endian.
- format = "<" + code
- expected = string_reverse(expected)
-
- # Pack work?
- got = pack(format, x)
- verify(got == expected,
- "'%s'-pack of %r gave %r, not %r" %
- (format, x, got, expected))
-
- # Unpack work?
- retrieved = unpack(format, got)[0]
- verify(x == retrieved,
- "'%s'-unpack of %r gave %r, not %r" %
- (format, got, retrieved, x))
+ self.fail("did not raise error for float coerce")
+
+ def test_isbigendian(self):
+ self.assertEqual((struct.pack('=i', 1)[0] == chr(0)), ISBIGENDIAN)
- # Adding any byte should cause a "too big" error.
- any_err(unpack, format, '\x01' + got)
+ def test_consistence(self):
+ self.assertRaises(struct.error, struct.calcsize, 'Z')
+ sz = struct.calcsize('i')
+ self.assertEqual(sz * 3, struct.calcsize('iii'))
+
+ fmt = 'cbxxxxxxhhhhiillffd?'
+ fmt3 = '3c3b18x12h6i6l6f3d3?'
+ sz = struct.calcsize(fmt)
+ sz3 = struct.calcsize(fmt3)
+ self.assertEqual(sz * 3, sz3)
+
+ self.assertRaises(struct.error, struct.pack, 'iii', 3)
+ self.assertRaises(struct.error, struct.pack, 'i', 3, 3, 3)
+ self.assertRaises(struct.error, struct.pack, 'i', 'foo')
+ self.assertRaises(struct.error, struct.pack, 'P', 'foo')
+ self.assertRaises(struct.error, struct.unpack, 'd', 'flap')
+ s = struct.pack('ii', 1, 2)
+ self.assertRaises(struct.error, struct.unpack, 'iii', s)
+ self.assertRaises(struct.error, struct.unpack, 'i', s)
+
+ def test_transitiveness(self):
+ c = 'a'
+ b = 1
+ h = 255
+ i = 65535
+ l = 65536
+ f = 3.1415
+ d = 3.1415
+ t = True
+
+ for prefix in ('', '@', '<', '>', '=', '!'):
+ for format in ('xcbhilfd?', 'xcBHILfd?'):
+ format = prefix + format
+ s = struct.pack(format, c, b, h, i, l, f, d, t)
+ cp, bp, hp, ip, lp, fp, dp, tp = struct.unpack(format, s)
+ self.assertEqual(cp, c)
+ self.assertEqual(bp, b)
+ self.assertEqual(hp, h)
+ self.assertEqual(ip, i)
+ self.assertEqual(lp, l)
+ self.assertEqual(int(100 * fp), int(100 * f))
+ self.assertEqual(int(100 * dp), int(100 * d))
+ self.assertEqual(tp, t)
+
+ def test_new_features(self):
+ # Test some of the new features in detail
+ # (format, argument, big-endian result, little-endian result, asymmetric)
+ tests = [
+ ('c', 'a', 'a', 'a', 0),
+ ('xc', 'a', '\0a', '\0a', 0),
+ ('cx', 'a', 'a\0', 'a\0', 0),
+ ('s', 'a', 'a', 'a', 0),
+ ('0s', 'helloworld', '', '', 1),
+ ('1s', 'helloworld', 'h', 'h', 1),
+ ('9s', 'helloworld', 'helloworl', 'helloworl', 1),
+ ('10s', 'helloworld', 'helloworld', 'helloworld', 0),
+ ('11s', 'helloworld', 'helloworld\0', 'helloworld\0', 1),
+ ('20s', 'helloworld', 'helloworld'+10*'\0', 'helloworld'+10*'\0', 1),
+ ('b', 7, '\7', '\7', 0),
+ ('b', -7, '\371', '\371', 0),
+ ('B', 7, '\7', '\7', 0),
+ ('B', 249, '\371', '\371', 0),
+ ('h', 700, '\002\274', '\274\002', 0),
+ ('h', -700, '\375D', 'D\375', 0),
+ ('H', 700, '\002\274', '\274\002', 0),
+ ('H', 0x10000-700, '\375D', 'D\375', 0),
+ ('i', 70000000, '\004,\035\200', '\200\035,\004', 0),
+ ('i', -70000000, '\373\323\342\200', '\200\342\323\373', 0),
+ ('I', 70000000L, '\004,\035\200', '\200\035,\004', 0),
+ ('I', 0x100000000L-70000000, '\373\323\342\200', '\200\342\323\373', 0),
+ ('l', 70000000, '\004,\035\200', '\200\035,\004', 0),
+ ('l', -70000000, '\373\323\342\200', '\200\342\323\373', 0),
+ ('L', 70000000L, '\004,\035\200', '\200\035,\004', 0),
+ ('L', 0x100000000L-70000000, '\373\323\342\200', '\200\342\323\373', 0),
+ ('f', 2.0, '@\000\000\000', '\000\000\000@', 0),
+ ('d', 2.0, '@\000\000\000\000\000\000\000',
+ '\000\000\000\000\000\000\000@', 0),
+ ('f', -2.0, '\300\000\000\000', '\000\000\000\300', 0),
+ ('d', -2.0, '\300\000\000\000\000\000\000\000',
+ '\000\000\000\000\000\000\000\300', 0),
+ ('?', 0, '\0', '\0', 0),
+ ('?', 3, '\1', '\1', 1),
+ ('?', True, '\1', '\1', 0),
+ ('?', [], '\0', '\0', 1),
+ ('?', (1,), '\1', '\1', 1),
+ ]
+
+ for fmt, arg, big, lil, asy in tests:
+ for (xfmt, exp) in [('>'+fmt, big), ('!'+fmt, big), ('<'+fmt, lil),
+ ('='+fmt, ISBIGENDIAN and big or lil)]:
+ res = struct.pack(xfmt, arg)
+ self.assertEqual(res, exp)
+ self.assertEqual(struct.calcsize(xfmt), len(res))
+ rev = struct.unpack(xfmt, res)[0]
+ if rev != arg:
+ self.assert_(asy)
+
+ def test_native_qQ(self):
+ # can't pack -1 as unsigned regardless
+ self.assertRaises((struct.error, TypeError), struct.pack, "Q", -1)
+ # can't pack string as 'q' regardless
+ self.assertRaises(struct.error, struct.pack, "q", "a")
+ # ditto, but 'Q'
+ self.assertRaises(struct.error, struct.pack, "Q", "a")
+
+ try:
+ struct.pack("q", 5)
+ except struct.error:
+ # does not have native q/Q
+ pass
else:
- # x is out of range -- verify pack realizes that.
- if not PY_STRUCT_RANGE_CHECKING and code in self.BUGGY_RANGE_CHECK:
- if verbose:
- print "Skipping buggy range check for code", code
- else:
- deprecated_err(pack, ">" + code, x)
- deprecated_err(pack, "<" + code, x)
-
- def run(self):
- from random import randrange
-
- # Create all interesting powers of 2.
- values = []
- for exp in range(self.bitsize + 3):
- values.append(1L << exp)
-
- # Add some random values.
- for i in range(self.bitsize):
- val = 0L
- for j in range(self.bytesize):
- val = (val << 8) | randrange(256)
- values.append(val)
-
- # Try all those, and their negations, and +-1 from them. Note
- # that this tests all power-of-2 boundaries in range, and a few out
- # of range, plus +-(2**n +- 1).
- for base in values:
- for val in -base, base:
- for incr in -1, 0, 1:
- x = val + incr
- try:
- x = int(x)
- except OverflowError:
- pass
- self.test_one(x)
-
- # Some error cases.
- for direction in "<>":
- for code in self.formatpair:
- for badobject in "a string", 3+42j, randrange:
- any_err(struct.pack, direction + code, badobject)
-
-for args in [("bB", 1),
- ("hH", 2),
- ("iI", 4),
- ("lL", 4),
- ("qQ", 8)]:
- t = IntTester(*args)
- t.run()
-
-
-###########################################################################
-# The p ("Pascal string") code.
-
-def test_p_code():
- for code, input, expected, expectedback in [
- ('p','abc', '\x00', ''),
- ('1p', 'abc', '\x00', ''),
- ('2p', 'abc', '\x01a', 'a'),
- ('3p', 'abc', '\x02ab', 'ab'),
- ('4p', 'abc', '\x03abc', 'abc'),
- ('5p', 'abc', '\x03abc\x00', 'abc'),
- ('6p', 'abc', '\x03abc\x00\x00', 'abc'),
- ('1000p', 'x'*1000, '\xff' + 'x'*999, 'x'*255)]:
- got = struct.pack(code, input)
- if got != expected:
- raise TestFailed("pack(%r, %r) == %r but expected %r" %
- (code, input, got, expected))
- (got,) = struct.unpack(code, got)
- if got != expectedback:
- raise TestFailed("unpack(%r, %r) == %r but expected %r" %
- (code, input, got, expectedback))
-
-test_p_code()
-
-
-###########################################################################
-# SF bug 705836. "<f" and ">f" had a severe rounding bug, where a carry
-# from the low-order discarded bits could propagate into the exponent
-# field, causing the result to be wrong by a factor of 2.
-
-def test_705836():
- import math
-
- for base in range(1, 33):
- # smaller <- largest representable float less than base.
- delta = 0.5
- while base - delta / 2.0 != base:
- delta /= 2.0
- smaller = base - delta
- # Packing this rounds away a solid string of trailing 1 bits.
- packed = struct.pack("<f", smaller)
- unpacked = struct.unpack("<f", packed)[0]
- # This failed at base = 2, 4, and 32, with unpacked = 1, 2, and
- # 16, respectively.
- verify(base == unpacked)
- bigpacked = struct.pack(">f", smaller)
- verify(bigpacked == string_reverse(packed),
- ">f pack should be byte-reversal of <f pack")
- unpacked = struct.unpack(">f", bigpacked)[0]
- verify(base == unpacked)
-
- # Largest finite IEEE single.
- big = (1 << 24) - 1
- big = math.ldexp(big, 127 - 23)
- packed = struct.pack(">f", big)
- unpacked = struct.unpack(">f", packed)[0]
- verify(big == unpacked)
-
- # The same, but tack on a 1 bit so it rounds up to infinity.
- big = (1 << 25) - 1
- big = math.ldexp(big, 127 - 24)
- try:
+ bytes = struct.calcsize('q')
+ # The expected values here are in big-endian format, primarily
+ # because I'm on a little-endian machine and so this is the
+ # clearest way (for me) to force the code to get exercised.
+ for format, input, expected in (
+ ('q', -1, '\xff' * bytes),
+ ('q', 0, '\x00' * bytes),
+ ('Q', 0, '\x00' * bytes),
+ ('q', 1L, '\x00' * (bytes-1) + '\x01'),
+ ('Q', (1L << (8*bytes))-1, '\xff' * bytes),
+ ('q', (1L << (8*bytes-1))-1, '\x7f' + '\xff' * (bytes - 1))):
+ got = struct.pack(format, input)
+ native_expected = bigendian_to_native(expected)
+ self.assertEqual(got, native_expected)
+ retrieved = struct.unpack(format, got)[0]
+ self.assertEqual(retrieved, input)
+
+ def test_standard_integers(self):
+ # Standard integer tests (bBhHiIlLqQ).
+ import binascii
+
+ class IntTester(unittest.TestCase):
+
+ # XXX Most std integer modes fail to test for out-of-range.
+ # The "i" and "l" codes appear to range-check OK on 32-bit boxes, but
+ # fail to check correctly on some 64-bit ones (Tru64 Unix + Compaq C
+ # reported by Mark Favas).
+ BUGGY_RANGE_CHECK = "bBhHiIlL"
+
+ def __init__(self, formatpair, bytesize):
+ self.assertEqual(len(formatpair), 2)
+ self.formatpair = formatpair
+ for direction in "<>!=":
+ for code in formatpair:
+ format = direction + code
+ self.assertEqual(struct.calcsize(format), bytesize)
+ self.bytesize = bytesize
+ self.bitsize = bytesize * 8
+ self.signed_code, self.unsigned_code = formatpair
+ self.unsigned_min = 0
+ self.unsigned_max = 2L**self.bitsize - 1
+ self.signed_min = -(2L**(self.bitsize-1))
+ self.signed_max = 2L**(self.bitsize-1) - 1
+
+ def test_one(self, x, pack=struct.pack,
+ unpack=struct.unpack,
+ unhexlify=binascii.unhexlify):
+ # Try signed.
+ code = self.signed_code
+ if self.signed_min <= x <= self.signed_max:
+ # Try big-endian.
+ expected = long(x)
+ if x < 0:
+ expected += 1L << self.bitsize
+ self.assert_(expected > 0)
+ expected = hex(expected)[2:-1] # chop "0x" and trailing 'L'
+ if len(expected) & 1:
+ expected = "0" + expected
+ expected = unhexlify(expected)
+ expected = "\x00" * (self.bytesize - len(expected)) + expected
+
+ # Pack work?
+ format = ">" + code
+ got = pack(format, x)
+ self.assertEqual(got, expected)
+
+ # Unpack work?
+ retrieved = unpack(format, got)[0]
+ self.assertEqual(x, retrieved)
+
+ # Adding any byte should cause a "too big" error.
+ self.assertRaises((struct.error, TypeError), unpack, format,
+ '\x01' + got)
+ # Try little-endian.
+ format = "<" + code
+ expected = string_reverse(expected)
+
+ # Pack work?
+ got = pack(format, x)
+ self.assertEqual(got, expected)
+
+ # Unpack work?
+ retrieved = unpack(format, got)[0]
+ self.assertEqual(x, retrieved)
+
+ # Adding any byte should cause a "too big" error.
+ self.assertRaises((struct.error, TypeError), unpack, format,
+ '\x01' + got)
+
+ else:
+ # x is out of range -- verify pack realizes that.
+ if not PY_STRUCT_RANGE_CHECKING and code in self.BUGGY_RANGE_CHECK:
+ if verbose:
+ print "Skipping buggy range check for code", code
+ else:
+ deprecated_err(pack, ">" + code, x)
+ deprecated_err(pack, "<" + code, x)
+
+ # Much the same for unsigned.
+ code = self.unsigned_code
+ if self.unsigned_min <= x <= self.unsigned_max:
+ # Try big-endian.
+ format = ">" + code
+ expected = long(x)
+ expected = hex(expected)[2:-1] # chop "0x" and trailing 'L'
+ if len(expected) & 1:
+ expected = "0" + expected
+ expected = unhexlify(expected)
+ expected = "\x00" * (self.bytesize - len(expected)) + expected
+
+ # Pack work?
+ got = pack(format, x)
+ self.assertEqual(got, expected)
+
+ # Unpack work?
+ retrieved = unpack(format, got)[0]
+ self.assertEqual(x, retrieved)
+
+ # Adding any byte should cause a "too big" error.
+ self.assertRaises((struct.error, TypeError), unpack, format,
+ '\x01' + got)
+
+ # Try little-endian.
+ format = "<" + code
+ expected = string_reverse(expected)
+
+ # Pack work?
+ got = pack(format, x)
+ self.assertEqual(got, expected)
+
+ # Unpack work?
+ retrieved = unpack(format, got)[0]
+ self.assertEqual(x, retrieved)
+
+ # Adding any byte should cause a "too big" error.
+ self.assertRaises((struct.error, TypeError), unpack, format,
+ '\x01' + got)
+
+ else:
+ # x is out of range -- verify pack realizes that.
+ if not PY_STRUCT_RANGE_CHECKING and code in self.BUGGY_RANGE_CHECK:
+ if verbose:
+ print "Skipping buggy range check for code", code
+ else:
+ deprecated_err(pack, ">" + code, x)
+ deprecated_err(pack, "<" + code, x)
+
+ def run(self):
+ from random import randrange
+
+ # Create all interesting powers of 2.
+ values = []
+ for exp in range(self.bitsize + 3):
+ values.append(1L << exp)
+
+ # Add some random values.
+ for i in range(self.bitsize):
+ val = 0L
+ for j in range(self.bytesize):
+ val = (val << 8) | randrange(256)
+ values.append(val)
+
+ # Try all those, and their negations, and +-1 from them. Note
+ # that this tests all power-of-2 boundaries in range, and a few out
+ # of range, plus +-(2**n +- 1).
+ for base in values:
+ for val in -base, base:
+ for incr in -1, 0, 1:
+ x = val + incr
+ try:
+ x = int(x)
+ except OverflowError:
+ pass
+ self.test_one(x)
+
+ # Some error cases.
+ for direction in "<>":
+ for code in self.formatpair:
+ for badobject in "a string", 3+42j, randrange:
+ self.assertRaises((struct.error, TypeError),
+ struct.pack, direction + code,
+ badobject)
+
+ for args in [("bB", 1),
+ ("hH", 2),
+ ("iI", 4),
+ ("lL", 4),
+ ("qQ", 8)]:
+ t = IntTester(*args)
+ t.run()
+
+ def test_p_code(self):
+ # Test p ("Pascal string") code.
+ for code, input, expected, expectedback in [
+ ('p','abc', '\x00', ''),
+ ('1p', 'abc', '\x00', ''),
+ ('2p', 'abc', '\x01a', 'a'),
+ ('3p', 'abc', '\x02ab', 'ab'),
+ ('4p', 'abc', '\x03abc', 'abc'),
+ ('5p', 'abc', '\x03abc\x00', 'abc'),
+ ('6p', 'abc', '\x03abc\x00\x00', 'abc'),
+ ('1000p', 'x'*1000, '\xff' + 'x'*999, 'x'*255)]:
+ got = struct.pack(code, input)
+ self.assertEqual(got, expected)
+ (got,) = struct.unpack(code, got)
+ self.assertEqual(got, expectedback)
+
+ def test_705836(self):
+ # SF bug 705836. "<f" and ">f" had a severe rounding bug, where a carry
+ # from the low-order discarded bits could propagate into the exponent
+ # field, causing the result to be wrong by a factor of 2.
+ import math
+
+ for base in range(1, 33):
+ # smaller <- largest representable float less than base.
+ delta = 0.5
+ while base - delta / 2.0 != base:
+ delta /= 2.0
+ smaller = base - delta
+ # Packing this rounds away a solid string of trailing 1 bits.
+ packed = struct.pack("<f", smaller)
+ unpacked = struct.unpack("<f", packed)[0]
+ # This failed at base = 2, 4, and 32, with unpacked = 1, 2, and
+ # 16, respectively.
+ self.assertEqual(base, unpacked)
+ bigpacked = struct.pack(">f", smaller)
+ self.assertEqual(bigpacked, string_reverse(packed))
+ unpacked = struct.unpack(">f", bigpacked)[0]
+ self.assertEqual(base, unpacked)
+
+ # Largest finite IEEE single.
+ big = (1 << 24) - 1
+ big = math.ldexp(big, 127 - 23)
packed = struct.pack(">f", big)
- except OverflowError:
- pass
- else:
- raise TestFailed("expected OverflowError")
-
-test_705836()
-
-###########################################################################
-# SF bug 1229380. No struct.pack exception for some out of range integers
+ unpacked = struct.unpack(">f", packed)[0]
+ self.assertEqual(big, unpacked)
-def test_1229380():
- import sys
- for endian in ('', '>', '<'):
- for cls in (int, long):
- for fmt in ('B', 'H', 'I', 'L'):
- deprecated_err(struct.pack, endian + fmt, cls(-1))
-
- deprecated_err(struct.pack, endian + 'B', cls(300))
- deprecated_err(struct.pack, endian + 'H', cls(70000))
-
- deprecated_err(struct.pack, endian + 'I', sys.maxint * 4L)
- deprecated_err(struct.pack, endian + 'L', sys.maxint * 4L)
-
-if PY_STRUCT_RANGE_CHECKING:
- test_1229380()
-
-###########################################################################
-# SF bug 1530559. struct.pack raises TypeError where it used to convert.
-
-def check_float_coerce(format, number):
- if PY_STRUCT_FLOAT_COERCE == 2:
- # Test for pre-2.5 struct module
- packed = struct.pack(format, number)
- floored = struct.unpack(format, packed)[0]
- if floored != int(number):
- raise TestFailed("did not correcly coerce float to int")
- return
- try:
- func(*args)
- except (struct.error, TypeError):
- if PY_STRUCT_FLOAT_COERCE:
- raise TestFailed("expected DeprecationWarning for float coerce")
- except DeprecationWarning:
- if not PY_STRUCT_FLOAT_COERCE:
- raise TestFailed("expected to raise struct.error for float coerce")
- else:
- raise TestFailed("did not raise error for float coerce")
-
-check_float_coerce = with_warning_restore(deprecated_err)
-
-def test_1530559():
- for endian in ('', '>', '<'):
- for fmt in ('B', 'H', 'I', 'L', 'b', 'h', 'i', 'l'):
- check_float_coerce(endian + fmt, 1.0)
- check_float_coerce(endian + fmt, 1.5)
-
-test_1530559()
+ # The same, but tack on a 1 bit so it rounds up to infinity.
+ big = (1 << 25) - 1
+ big = math.ldexp(big, 127 - 24)
+ self.assertRaises(OverflowError, struct.pack, ">f", big)
+
+ if PY_STRUCT_RANGE_CHECKING:
+ def test_1229380(self):
+ # SF bug 1229380. No struct.pack exception for some out of
+ # range integers
+ import sys
+ for endian in ('', '>', '<'):
+ for cls in (int, long):
+ for fmt in ('B', 'H', 'I', 'L'):
+ deprecated_err(struct.pack, endian + fmt, cls(-1))
+
+ deprecated_err(struct.pack, endian + 'B', cls(300))
+ deprecated_err(struct.pack, endian + 'H', cls(70000))
+
+ deprecated_err(struct.pack, endian + 'I', sys.maxint * 4L)
+ deprecated_err(struct.pack, endian + 'L', sys.maxint * 4L)
+
+ def XXXtest_1530559(self):
+ # XXX This is broken: see the bug report
+ # SF bug 1530559. struct.pack raises TypeError where it used to convert.
+ for endian in ('', '>', '<'):
+ for fmt in ('B', 'H', 'I', 'L', 'b', 'h', 'i', 'l'):
+ self.check_float_coerce(endian + fmt, 1.0)
+ self.check_float_coerce(endian + fmt, 1.5)
+
+ def test_unpack_from(self):
+ test_string = 'abcd01234'
+ fmt = '4s'
+ s = struct.Struct(fmt)
+ for cls in (str, buffer):
+ data = cls(test_string)
+ self.assertEqual(s.unpack_from(data), ('abcd',))
+ self.assertEqual(s.unpack_from(data, 2), ('cd01',))
+ self.assertEqual(s.unpack_from(data, 4), ('0123',))
+ for i in xrange(6):
+ self.assertEqual(s.unpack_from(data, i), (data[i:i+4],))
+ for i in xrange(6, len(test_string) + 1):
+ self.assertRaises(struct.error, s.unpack_from, data, i)
+ for cls in (str, buffer):
+ data = cls(test_string)
+ self.assertEqual(struct.unpack_from(fmt, data), ('abcd',))
+ self.assertEqual(struct.unpack_from(fmt, data, 2), ('cd01',))
+ self.assertEqual(struct.unpack_from(fmt, data, 4), ('0123',))
+ for i in xrange(6):
+ self.assertEqual(struct.unpack_from(fmt, data, i), (data[i:i+4],))
+ for i in xrange(6, len(test_string) + 1):
+ self.assertRaises(struct.error, struct.unpack_from, fmt, data, i)
+
+ def test_pack_into(self):
+ test_string = 'Reykjavik rocks, eow!'
+ writable_buf = array.array('c', ' '*100)
+ fmt = '21s'
+ s = struct.Struct(fmt)
+
+ # Test without offset
+ s.pack_into(writable_buf, 0, test_string)
+ from_buf = writable_buf.tostring()[:len(test_string)]
+ self.assertEqual(from_buf, test_string)
+
+ # Test with offset.
+ s.pack_into(writable_buf, 10, test_string)
+ from_buf = writable_buf.tostring()[:len(test_string)+10]
+ self.assertEqual(from_buf, test_string[:10] + test_string)
+
+ # Go beyond boundaries.
+ small_buf = array.array('c', ' '*10)
+ self.assertRaises(struct.error, s.pack_into, small_buf, 0, test_string)
+ self.assertRaises(struct.error, s.pack_into, small_buf, 2, test_string)
+
+ def test_pack_into_fn(self):
+ test_string = 'Reykjavik rocks, eow!'
+ writable_buf = array.array('c', ' '*100)
+ fmt = '21s'
+ pack_into = lambda *args: struct.pack_into(fmt, *args)
+
+ # Test without offset.
+ pack_into(writable_buf, 0, test_string)
+ from_buf = writable_buf.tostring()[:len(test_string)]
+ self.assertEqual(from_buf, test_string)
+
+ # Test with offset.
+ pack_into(writable_buf, 10, test_string)
+ from_buf = writable_buf.tostring()[:len(test_string)+10]
+ self.assertEqual(from_buf, test_string[:10] + test_string)
+
+ # Go beyond boundaries.
+ small_buf = array.array('c', ' '*10)
+ self.assertRaises(struct.error, pack_into, small_buf, 0, test_string)
+ self.assertRaises(struct.error, pack_into, small_buf, 2, test_string)
+
+ def test_unpack_with_buffer(self):
+ # SF bug 1563759: struct.unpack doens't support buffer protocol objects
+ data1 = array.array('B', '\x12\x34\x56\x78')
+ data2 = buffer('......\x12\x34\x56\x78......', 6, 4)
+ for data in [data1, data2]:
+ value, = struct.unpack('>I', data)
+ self.assertEqual(value, 0x12345678)
+
+ def test_bool(self):
+ for prefix in tuple("<>!=")+('',):
+ false = (), [], [], '', 0
+ true = [1], 'test', 5, -1, 0xffffffffL+1, 0xffffffff/2
+
+ falseFormat = prefix + '?' * len(false)
+ packedFalse = struct.pack(falseFormat, *false)
+ unpackedFalse = struct.unpack(falseFormat, packedFalse)
+
+ trueFormat = prefix + '?' * len(true)
+ packedTrue = struct.pack(trueFormat, *true)
+ unpackedTrue = struct.unpack(trueFormat, packedTrue)
+
+ self.assertEqual(len(true), len(unpackedTrue))
+ self.assertEqual(len(false), len(unpackedFalse))
+
+ for t in unpackedFalse:
+ self.assertFalse(t)
+ for t in unpackedTrue:
+ self.assertTrue(t)
+
+ packed = struct.pack(prefix+'?', 1)
+
+ self.assertEqual(len(packed), struct.calcsize(prefix+'?'))
+
+ if len(packed) != 1:
+ self.assertFalse(prefix, msg='encoded bool is not one byte: %r'
+ %packed)
+
+ for c in '\x01\x7f\xff\x0f\xf0':
+ self.assertTrue(struct.unpack('>?', c)[0])
+
+ def test_crasher(self):
+ if IS32BIT:
+ self.assertRaises(MemoryError, struct.pack, "357913941c", "a")
+ else:
+ print "%s test_crasher skipped on 64bit build."
-###########################################################################
-# Packing and unpacking to/from buffers.
-# Copied and modified from unittest.
-def assertRaises(excClass, callableObj, *args, **kwargs):
- try:
- callableObj(*args, **kwargs)
- except excClass:
- return
- else:
- raise TestFailed("%s not raised." % excClass)
-def test_unpack_from():
- test_string = 'abcd01234'
- fmt = '4s'
- s = struct.Struct(fmt)
- for cls in (str, buffer):
- data = cls(test_string)
- vereq(s.unpack_from(data), ('abcd',))
- vereq(s.unpack_from(data, 2), ('cd01',))
- vereq(s.unpack_from(data, 4), ('0123',))
- for i in xrange(6):
- vereq(s.unpack_from(data, i), (data[i:i+4],))
- for i in xrange(6, len(test_string) + 1):
- simple_err(s.unpack_from, data, i)
- for cls in (str, buffer):
- data = cls(test_string)
- vereq(struct.unpack_from(fmt, data), ('abcd',))
- vereq(struct.unpack_from(fmt, data, 2), ('cd01',))
- vereq(struct.unpack_from(fmt, data, 4), ('0123',))
- for i in xrange(6):
- vereq(struct.unpack_from(fmt, data, i), (data[i:i+4],))
- for i in xrange(6, len(test_string) + 1):
- simple_err(struct.unpack_from, fmt, data, i)
-
-def test_pack_into():
- test_string = 'Reykjavik rocks, eow!'
- writable_buf = array.array('c', ' '*100)
- fmt = '21s'
- s = struct.Struct(fmt)
-
- # Test without offset
- s.pack_into(writable_buf, 0, test_string)
- from_buf = writable_buf.tostring()[:len(test_string)]
- vereq(from_buf, test_string)
-
- # Test with offset.
- s.pack_into(writable_buf, 10, test_string)
- from_buf = writable_buf.tostring()[:len(test_string)+10]
- vereq(from_buf, test_string[:10] + test_string)
-
- # Go beyond boundaries.
- small_buf = array.array('c', ' '*10)
- assertRaises(struct.error, s.pack_into, small_buf, 0, test_string)
- assertRaises(struct.error, s.pack_into, small_buf, 2, test_string)
-
-def test_pack_into_fn():
- test_string = 'Reykjavik rocks, eow!'
- writable_buf = array.array('c', ' '*100)
- fmt = '21s'
- pack_into = lambda *args: struct.pack_into(fmt, *args)
-
- # Test without offset.
- pack_into(writable_buf, 0, test_string)
- from_buf = writable_buf.tostring()[:len(test_string)]
- vereq(from_buf, test_string)
-
- # Test with offset.
- pack_into(writable_buf, 10, test_string)
- from_buf = writable_buf.tostring()[:len(test_string)+10]
- vereq(from_buf, test_string[:10] + test_string)
-
- # Go beyond boundaries.
- small_buf = array.array('c', ' '*10)
- assertRaises(struct.error, pack_into, small_buf, 0, test_string)
- assertRaises(struct.error, pack_into, small_buf, 2, test_string)
-
-def test_unpack_with_buffer():
- # SF bug 1563759: struct.unpack doens't support buffer protocol objects
- data1 = array.array('B', '\x12\x34\x56\x78')
- data2 = buffer('......\x12\x34\x56\x78......', 6, 4)
- for data in [data1, data2]:
- value, = struct.unpack('>I', data)
- vereq(value, 0x12345678)
-
-# Test methods to pack and unpack from buffers rather than strings.
-test_unpack_from()
-test_pack_into()
-test_pack_into_fn()
-test_unpack_with_buffer()
-
-def test_bool():
- for prefix in tuple("<>!=")+('',):
- false = (), [], [], '', 0
- true = [1], 'test', 5, -1, 0xffffffffL+1, 0xffffffff/2
-
- falseFormat = prefix + '?' * len(false)
- if verbose:
- print 'trying bool pack/unpack on', false, 'using format', falseFormat
- packedFalse = struct.pack(falseFormat, *false)
- unpackedFalse = struct.unpack(falseFormat, packedFalse)
-
- trueFormat = prefix + '?' * len(true)
- if verbose:
- print 'trying bool pack/unpack on', true, 'using format', trueFormat
- packedTrue = struct.pack(trueFormat, *true)
- unpackedTrue = struct.unpack(trueFormat, packedTrue)
-
- if len(true) != len(unpackedTrue):
- raise TestFailed('unpacked true array is not of same size as input')
- if len(false) != len(unpackedFalse):
- raise TestFailed('unpacked false array is not of same size as input')
-
- for t in unpackedFalse:
- if t is not False:
- raise TestFailed('%r did not unpack as False' % t)
- for t in unpackedTrue:
- if t is not True:
- raise TestFailed('%r did not unpack as false' % t)
-
- if prefix and verbose:
- print 'trying size of bool with format %r' % (prefix+'?')
- packed = struct.pack(prefix+'?', 1)
-
- if len(packed) != struct.calcsize(prefix+'?'):
- raise TestFailed('packed length is not equal to calculated size')
-
- if len(packed) != 1 and prefix:
- raise TestFailed('encoded bool is not one byte: %r' % packed)
- elif not prefix and verbose:
- print 'size of bool in native format is %i' % (len(packed))
-
- for c in '\x01\x7f\xff\x0f\xf0':
- if struct.unpack('>?', c)[0] is not True:
- raise TestFailed('%c did not unpack as True' % c)
+def test_main():
+ run_unittest(StructTest)
-test_bool()
+if __name__ == '__main__':
+ test_main()
Modified: python/branches/tlee-ast-optimize/Makefile.pre.in
==============================================================================
--- python/branches/tlee-ast-optimize/Makefile.pre.in (original)
+++ python/branches/tlee-ast-optimize/Makefile.pre.in Fri Jun 13 16:43:12 2008
@@ -304,7 +304,6 @@
Objects/bufferobject.o \
Objects/bytes_methods.o \
Objects/bytearrayobject.o \
- Objects/bytesobject.o \
Objects/cellobject.o \
Objects/classobject.o \
Objects/cobject.o \
@@ -328,8 +327,9 @@
Objects/object.o \
Objects/obmalloc.o \
Objects/rangeobject.o \
- Objects/setobject.o \
+ Objects/setobject.o \
Objects/sliceobject.o \
+ Objects/stringobject.o \
Objects/structseq.o \
Objects/tupleobject.o \
Objects/typeobject.o \
@@ -564,8 +564,6 @@
Objects/stringobject.o: $(srcdir)/Objects/stringobject.c \
$(STRINGLIB_HEADERS)
-Objects/bytesobject.o: $(srcdir)/Objects/bytesobject.c \
- $(STRINGLIB_HEADERS)
Python/formatter_unicode.o: $(srcdir)/Python/formatter_unicode.c \
$(STRINGLIB_HEADERS)
Modified: python/branches/tlee-ast-optimize/Misc/NEWS
==============================================================================
--- python/branches/tlee-ast-optimize/Misc/NEWS (original)
+++ python/branches/tlee-ast-optimize/Misc/NEWS Fri Jun 13 16:43:12 2008
@@ -13,7 +13,7 @@
-----------------
- Several set methods now accept multiple arguments: update(), union(),
- intersection() and intersection_update().
+ intersection(), intersection_update(), difference(), and difference_update().
- Issue #2898: Added sys.getsizeof() to retrieve size of objects in bytes.
@@ -40,6 +40,11 @@
Exception (KeyboardInterrupt, and SystemExit) propagate instead of
ignoring them.
+- Added checks for integer overflows, contributed by Google. Some are
+ only available if asserts are left in the code, in cases where they
+ can't be triggered from Python code.
+
+
Extension Modules
-----------------
Modified: python/branches/tlee-ast-optimize/Modules/_csv.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/_csv.c (original)
+++ python/branches/tlee-ast-optimize/Modules/_csv.c Fri Jun 13 16:43:12 2008
@@ -559,6 +559,10 @@
self->field = PyMem_Malloc(self->field_size);
}
else {
+ if (self->field_size > INT_MAX / 2) {
+ PyErr_NoMemory();
+ return 0;
+ }
self->field_size *= 2;
self->field = PyMem_Realloc(self->field, self->field_size);
}
@@ -1053,6 +1057,12 @@
static int
join_check_rec_size(WriterObj *self, int rec_len)
{
+
+ if (rec_len < 0 || rec_len > INT_MAX - MEM_INCR) {
+ PyErr_NoMemory();
+ return 0;
+ }
+
if (rec_len > self->rec_size) {
if (self->rec_size == 0) {
self->rec_size = (rec_len / MEM_INCR + 1) * MEM_INCR;
Modified: python/branches/tlee-ast-optimize/Modules/_heapqmodule.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/_heapqmodule.c (original)
+++ python/branches/tlee-ast-optimize/Modules/_heapqmodule.c Fri Jun 13 16:43:12 2008
@@ -17,13 +17,18 @@
cmp_lt(PyObject *x, PyObject *y)
{
int cmp;
- cmp = PyObject_RichCompareBool(x, y, Py_LT);
- if (cmp == -1 && PyErr_ExceptionMatches(PyExc_AttributeError)) {
- PyErr_Clear();
- cmp = PyObject_RichCompareBool(y, x, Py_LE);
- if (cmp != -1)
- cmp = 1 - cmp;
+ static PyObject *lt = NULL;
+
+ if (lt == NULL) {
+ lt = PyString_FromString("__lt__");
+ if (lt == NULL)
+ return -1;
}
+ if (PyObject_HasAttr(x, lt))
+ return PyObject_RichCompareBool(x, y, Py_LT);
+ cmp = PyObject_RichCompareBool(y, x, Py_LE);
+ if (cmp != -1)
+ cmp = 1 - cmp;
return cmp;
}
Modified: python/branches/tlee-ast-optimize/Modules/_struct.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/_struct.c (original)
+++ python/branches/tlee-ast-optimize/Modules/_struct.c Fri Jun 13 16:43:12 2008
@@ -1385,6 +1385,12 @@
}
}
+ /* check for overflow */
+ if ((len + 1) > (PY_SSIZE_T_MAX / sizeof(formatcode))) {
+ PyErr_NoMemory();
+ return -1;
+ }
+
self->s_size = size;
self->s_len = len;
codes = PyMem_MALLOC((len + 1) * sizeof(formatcode));
Modified: python/branches/tlee-ast-optimize/Modules/arraymodule.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/arraymodule.c (original)
+++ python/branches/tlee-ast-optimize/Modules/arraymodule.c Fri Jun 13 16:43:12 2008
@@ -652,6 +652,9 @@
PyErr_BadArgument();
return NULL;
}
+ if (Py_SIZE(a) > PY_SSIZE_T_MAX - Py_SIZE(b)) {
+ return PyErr_NoMemory();
+ }
size = Py_SIZE(a) + Py_SIZE(b);
np = (arrayobject *) newarrayobject(&Arraytype, size, a->ob_descr);
if (np == NULL) {
@@ -674,6 +677,9 @@
Py_ssize_t nbytes;
if (n < 0)
n = 0;
+ if ((Py_SIZE(a) != 0) && (n > PY_SSIZE_T_MAX / Py_SIZE(a))) {
+ return PyErr_NoMemory();
+ }
size = Py_SIZE(a) * n;
np = (arrayobject *) newarrayobject(&Arraytype, size, a->ob_descr);
if (np == NULL)
@@ -818,6 +824,11 @@
"can only extend with array of same kind");
return -1;
}
+ if ((Py_SIZE(self) > PY_SSIZE_T_MAX - Py_SIZE(b)) ||
+ ((Py_SIZE(self) + Py_SIZE(b)) > PY_SSIZE_T_MAX / self->ob_descr->itemsize)) {
+ PyErr_NoMemory();
+ return -1;
+ }
size = Py_SIZE(self) + Py_SIZE(b);
PyMem_RESIZE(self->ob_item, char, size*self->ob_descr->itemsize);
if (self->ob_item == NULL) {
@@ -859,6 +870,10 @@
if (n < 0)
n = 0;
items = self->ob_item;
+ if ((self->ob_descr->itemsize != 0) &&
+ (Py_SIZE(self) > PY_SSIZE_T_MAX / self->ob_descr->itemsize)) {
+ return PyErr_NoMemory();
+ }
size = Py_SIZE(self) * self->ob_descr->itemsize;
if (n == 0) {
PyMem_FREE(items);
@@ -867,6 +882,9 @@
self->allocated = 0;
}
else {
+ if (size > PY_SSIZE_T_MAX / n) {
+ return PyErr_NoMemory();
+ }
PyMem_Resize(items, char, n * size);
if (items == NULL)
return PyErr_NoMemory();
@@ -1148,6 +1166,10 @@
Py_INCREF(dict);
}
if (Py_SIZE(array) > 0) {
+ if (array->ob_descr->itemsize
+ > PY_SSIZE_T_MAX / array->ob_size) {
+ return PyErr_NoMemory();
+ }
result = Py_BuildValue("O(cs#)O",
Py_TYPE(array),
array->ob_descr->typecode,
@@ -1330,6 +1352,9 @@
if ((*self->ob_descr->setitem)(self,
Py_SIZE(self) - n + i, v) != 0) {
Py_SIZE(self) -= n;
+ if (itemsize && (self->ob_size > PY_SSIZE_T_MAX / itemsize)) {
+ return PyErr_NoMemory();
+ }
PyMem_RESIZE(item, char,
Py_SIZE(self) * itemsize);
self->ob_item = item;
@@ -1389,6 +1414,10 @@
n = n / itemsize;
if (n > 0) {
char *item = self->ob_item;
+ if ((n > PY_SSIZE_T_MAX - Py_SIZE(self)) ||
+ ((Py_SIZE(self) + n) > PY_SSIZE_T_MAX / itemsize)) {
+ return PyErr_NoMemory();
+ }
PyMem_RESIZE(item, char, (Py_SIZE(self) + n) * itemsize);
if (item == NULL) {
PyErr_NoMemory();
@@ -1414,8 +1443,12 @@
static PyObject *
array_tostring(arrayobject *self, PyObject *unused)
{
- return PyString_FromStringAndSize(self->ob_item,
+ if (self->ob_size <= PY_SSIZE_T_MAX / self->ob_descr->itemsize) {
+ return PyString_FromStringAndSize(self->ob_item,
Py_SIZE(self) * self->ob_descr->itemsize);
+ } else {
+ return PyErr_NoMemory();
+ }
}
PyDoc_STRVAR(tostring_doc,
@@ -1443,6 +1476,9 @@
}
if (n > 0) {
Py_UNICODE *item = (Py_UNICODE *) self->ob_item;
+ if (Py_SIZE(self) > PY_SSIZE_T_MAX - n) {
+ return PyErr_NoMemory();
+ }
PyMem_RESIZE(item, Py_UNICODE, Py_SIZE(self) + n);
if (item == NULL) {
PyErr_NoMemory();
Modified: python/branches/tlee-ast-optimize/Modules/audioop.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/audioop.c (original)
+++ python/branches/tlee-ast-optimize/Modules/audioop.c Fri Jun 13 16:43:12 2008
@@ -829,7 +829,7 @@
audioop_tostereo(PyObject *self, PyObject *args)
{
signed char *cp, *ncp;
- int len, size, val1, val2, val = 0;
+ int len, new_len, size, val1, val2, val = 0;
double fac1, fac2, fval, maxval;
PyObject *rv;
int i;
@@ -846,7 +846,14 @@
return 0;
}
- rv = PyString_FromStringAndSize(NULL, len*2);
+ new_len = len*2;
+ if (new_len < 0) {
+ PyErr_SetString(PyExc_MemoryError,
+ "not enough memory for output buffer");
+ return 0;
+ }
+
+ rv = PyString_FromStringAndSize(NULL, new_len);
if ( rv == 0 )
return 0;
ncp = (signed char *)PyString_AsString(rv);
@@ -1009,7 +1016,7 @@
{
signed char *cp;
unsigned char *ncp;
- int len, size, size2, val = 0;
+ int len, new_len, size, size2, val = 0;
PyObject *rv;
int i, j;
@@ -1023,7 +1030,13 @@
return 0;
}
- rv = PyString_FromStringAndSize(NULL, (len/size)*size2);
+ new_len = (len/size)*size2;
+ if (new_len < 0) {
+ PyErr_SetString(PyExc_MemoryError,
+ "not enough memory for output buffer");
+ return 0;
+ }
+ rv = PyString_FromStringAndSize(NULL, new_len);
if ( rv == 0 )
return 0;
ncp = (unsigned char *)PyString_AsString(rv);
@@ -1059,6 +1072,7 @@
int chan, d, *prev_i, *cur_i, cur_o;
PyObject *state, *samps, *str, *rv = NULL;
int bytes_per_frame;
+ size_t alloc_size;
weightA = 1;
weightB = 0;
@@ -1101,8 +1115,14 @@
inrate /= d;
outrate /= d;
- prev_i = (int *) malloc(nchannels * sizeof(int));
- cur_i = (int *) malloc(nchannels * sizeof(int));
+ alloc_size = sizeof(int) * (unsigned)nchannels;
+ if (alloc_size < nchannels) {
+ PyErr_SetString(PyExc_MemoryError,
+ "not enough memory for output buffer");
+ return 0;
+ }
+ prev_i = (int *) malloc(alloc_size);
+ cur_i = (int *) malloc(alloc_size);
if (prev_i == NULL || cur_i == NULL) {
(void) PyErr_NoMemory();
goto exit;
@@ -1276,7 +1296,7 @@
unsigned char *cp;
unsigned char cval;
signed char *ncp;
- int len, size, val;
+ int len, new_len, size, val;
PyObject *rv;
int i;
@@ -1289,12 +1309,18 @@
return 0;
}
- rv = PyString_FromStringAndSize(NULL, len*size);
+ new_len = len*size;
+ if (new_len < 0) {
+ PyErr_SetString(PyExc_MemoryError,
+ "not enough memory for output buffer");
+ return 0;
+ }
+ rv = PyString_FromStringAndSize(NULL, new_len);
if ( rv == 0 )
return 0;
ncp = (signed char *)PyString_AsString(rv);
- for ( i=0; i < len*size; i += size ) {
+ for ( i=0; i < new_len; i += size ) {
cval = *cp++;
val = st_ulaw2linear16(cval);
@@ -1344,7 +1370,7 @@
unsigned char *cp;
unsigned char cval;
signed char *ncp;
- int len, size, val;
+ int len, new_len, size, val;
PyObject *rv;
int i;
@@ -1357,12 +1383,18 @@
return 0;
}
- rv = PyString_FromStringAndSize(NULL, len*size);
+ new_len = len*size;
+ if (new_len < 0) {
+ PyErr_SetString(PyExc_MemoryError,
+ "not enough memory for output buffer");
+ return 0;
+ }
+ rv = PyString_FromStringAndSize(NULL, new_len);
if ( rv == 0 )
return 0;
ncp = (signed char *)PyString_AsString(rv);
- for ( i=0; i < len*size; i += size ) {
+ for ( i=0; i < new_len; i += size ) {
cval = *cp++;
val = st_alaw2linear16(cval);
@@ -1487,7 +1519,7 @@
{
signed char *cp;
signed char *ncp;
- int len, size, valpred, step, delta, index, sign, vpdiff;
+ int len, new_len, size, valpred, step, delta, index, sign, vpdiff;
PyObject *rv, *str, *state;
int i, inputbuffer = 0, bufferstep;
@@ -1509,7 +1541,13 @@
} else if ( !PyArg_ParseTuple(state, "ii", &valpred, &index) )
return 0;
- str = PyString_FromStringAndSize(NULL, len*size*2);
+ new_len = len*size*2;
+ if (new_len < 0) {
+ PyErr_SetString(PyExc_MemoryError,
+ "not enough memory for output buffer");
+ return 0;
+ }
+ str = PyString_FromStringAndSize(NULL, new_len);
if ( str == 0 )
return 0;
ncp = (signed char *)PyString_AsString(str);
@@ -1517,7 +1555,7 @@
step = stepsizeTable[index];
bufferstep = 0;
- for ( i=0; i < len*size*2; i += size ) {
+ for ( i=0; i < new_len; i += size ) {
/* Step 1 - get the delta value and compute next index */
if ( bufferstep ) {
delta = inputbuffer & 0xf;
Modified: python/branches/tlee-ast-optimize/Modules/binascii.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/binascii.c (original)
+++ python/branches/tlee-ast-optimize/Modules/binascii.c Fri Jun 13 16:43:12 2008
@@ -141,7 +141,7 @@
#define BASE64_PAD '='
/* Max binary chunk size; limited only by available memory */
-#define BASE64_MAXBIN (INT_MAX/2 - sizeof(PyStringObject) - 3)
+#define BASE64_MAXBIN (PY_SSIZE_T_MAX/2 - sizeof(PyStringObject) - 3)
static unsigned char table_b2a_base64[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
@@ -198,6 +198,8 @@
if ( !PyArg_ParseTuple(args, "t#:a2b_uu", &ascii_data, &ascii_len) )
return NULL;
+ assert(ascii_len >= 0);
+
/* First byte: binary data length (in bytes) */
bin_len = (*ascii_data++ - ' ') & 077;
ascii_len--;
@@ -351,6 +353,11 @@
if ( !PyArg_ParseTuple(args, "t#:a2b_base64", &ascii_data, &ascii_len) )
return NULL;
+ assert(ascii_len >= 0);
+
+ if (ascii_len > PY_SSIZE_T_MAX - 3)
+ return PyErr_NoMemory();
+
bin_len = ((ascii_len+3)/4)*3; /* Upper bound, corrected later */
/* Allocate the buffer */
@@ -440,6 +447,9 @@
if ( !PyArg_ParseTuple(args, "s#:b2a_base64", &bin_data, &bin_len) )
return NULL;
+
+ assert(bin_len >= 0);
+
if ( bin_len > BASE64_MAXBIN ) {
PyErr_SetString(Error, "Too much data for base64 line");
return NULL;
@@ -495,6 +505,11 @@
if ( !PyArg_ParseTuple(args, "t#:a2b_hqx", &ascii_data, &len) )
return NULL;
+ assert(len >= 0);
+
+ if (len > PY_SSIZE_T_MAX - 2)
+ return PyErr_NoMemory();
+
/* Allocate a string that is too big (fixed later)
Add two to the initial length to prevent interning which
would preclude subsequent resizing. */
@@ -558,6 +573,11 @@
if ( !PyArg_ParseTuple(args, "s#:rlecode_hqx", &in_data, &len) )
return NULL;
+ assert(len >= 0);
+
+ if (len > PY_SSIZE_T_MAX / 2 - 2)
+ return PyErr_NoMemory();
+
/* Worst case: output is twice as big as input (fixed later) */
if ( (rv=PyString_FromStringAndSize(NULL, len*2+2)) == NULL )
return NULL;
@@ -607,6 +627,11 @@
if ( !PyArg_ParseTuple(args, "s#:b2a_hqx", &bin_data, &len) )
return NULL;
+ assert(len >= 0);
+
+ if (len > PY_SSIZE_T_MAX / 2 - 2)
+ return PyErr_NoMemory();
+
/* Allocate a buffer that is at least large enough */
if ( (rv=PyString_FromStringAndSize(NULL, len*2+2)) == NULL )
return NULL;
@@ -645,9 +670,13 @@
if ( !PyArg_ParseTuple(args, "s#:rledecode_hqx", &in_data, &in_len) )
return NULL;
+ assert(in_len >= 0);
+
/* Empty string is a special case */
if ( in_len == 0 )
return PyString_FromString("");
+ else if (in_len > PY_SSIZE_T_MAX / 2)
+ return PyErr_NoMemory();
/* Allocate a buffer of reasonable size. Resized when needed */
out_len = in_len*2;
@@ -673,6 +702,7 @@
#define OUTBYTE(b) \
do { \
if ( --out_len_left < 0 ) { \
+ if ( out_len > PY_SSIZE_T_MAX / 2) return PyErr_NoMemory(); \
_PyString_Resize(&rv, 2*out_len); \
if ( rv == NULL ) return NULL; \
out_data = (unsigned char *)PyString_AsString(rv) \
@@ -741,7 +771,7 @@
if ( !PyArg_ParseTuple(args, "s#i:crc_hqx", &bin_data, &len, &crc) )
return NULL;
- while(len--) {
+ while(len-- > 0) {
crc=((crc<<8)&0xff00)^crctab_hqx[((crc>>8)&0xff)^*bin_data++];
}
@@ -901,7 +931,7 @@
return NULL;
crc = ~ crc;
- while (len--)
+ while (len-- > 0)
crc = crc_32_tab[(crc ^ *bin_data++) & 0xffU] ^ (crc >> 8);
/* Note: (crc >> 8) MUST zero fill on left */
@@ -923,6 +953,10 @@
if (!PyArg_ParseTuple(args, "s#:b2a_hex", &argbuf, &arglen))
return NULL;
+ assert(arglen >= 0);
+ if (arglen > PY_SSIZE_T_MAX / 2)
+ return PyErr_NoMemory();
+
retval = PyString_FromStringAndSize(NULL, arglen*2);
if (!retval)
return NULL;
@@ -980,6 +1014,8 @@
if (!PyArg_ParseTuple(args, "s#:a2b_hex", &argbuf, &arglen))
return NULL;
+ assert(arglen >= 0);
+
/* XXX What should we do about strings with an odd length? Should
* we add an implicit leading zero, or a trailing zero? For now,
* raise an exception.
Modified: python/branches/tlee-ast-optimize/Modules/cPickle.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/cPickle.c (original)
+++ python/branches/tlee-ast-optimize/Modules/cPickle.c Fri Jun 13 16:43:12 2008
@@ -3435,6 +3435,14 @@
if (self->read_func(self, &s, 4) < 0) return -1;
l = calc_binint(s, 4);
+ if (l < 0) {
+ /* Corrupt or hostile pickle -- we never write one like
+ * this.
+ */
+ PyErr_SetString(UnpicklingError,
+ "BINSTRING pickle has negative byte count");
+ return -1;
+ }
if (self->read_func(self, &s, l) < 0)
return -1;
@@ -3502,6 +3510,14 @@
if (self->read_func(self, &s, 4) < 0) return -1;
l = calc_binint(s, 4);
+ if (l < 0) {
+ /* Corrupt or hostile pickle -- we never write one like
+ * this.
+ */
+ PyErr_SetString(UnpicklingError,
+ "BINUNICODE pickle has negative byte count");
+ return -1;
+ }
if (self->read_func(self, &s, l) < 0)
return -1;
Modified: python/branches/tlee-ast-optimize/Modules/cStringIO.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/cStringIO.c (original)
+++ python/branches/tlee-ast-optimize/Modules/cStringIO.c Fri Jun 13 16:43:12 2008
@@ -119,6 +119,7 @@
static PyObject *
IO_cgetval(PyObject *self) {
if (!IO__opencheck(IOOOBJECT(self))) return NULL;
+ assert(IOOOBJECT(self)->pos >= 0);
return PyString_FromStringAndSize(((IOobject*)self)->buf,
((IOobject*)self)->pos);
}
@@ -137,6 +138,7 @@
}
else
s=self->string_size;
+ assert(self->pos >= 0);
return PyString_FromStringAndSize(self->buf, s);
}
@@ -157,6 +159,8 @@
Py_ssize_t l;
if (!IO__opencheck(IOOOBJECT(self))) return -1;
+ assert(IOOOBJECT(self)->pos >= 0);
+ assert(IOOOBJECT(self)->string_size >= 0);
l = ((IOobject*)self)->string_size - ((IOobject*)self)->pos;
if (n < 0 || n > l) {
n = l;
@@ -192,12 +196,17 @@
for (n = ((IOobject*)self)->buf + ((IOobject*)self)->pos,
s = ((IOobject*)self)->buf + ((IOobject*)self)->string_size;
n < s && *n != '\n'; n++);
+
if (n < s) n++;
*output=((IOobject*)self)->buf + ((IOobject*)self)->pos;
l = n - ((IOobject*)self)->buf - ((IOobject*)self)->pos;
- assert(((IOobject*)self)->pos + l < INT_MAX);
- ((IOobject*)self)->pos += (int)l;
+
+ assert(IOOOBJECT(self)->pos <= PY_SSIZE_T_MAX - l);
+ assert(IOOOBJECT(self)->pos >= 0);
+ assert(IOOOBJECT(self)->string_size >= 0);
+
+ ((IOobject*)self)->pos += l;
return (int)l;
}
@@ -215,6 +224,7 @@
n -= m;
self->pos -= m;
}
+ assert(IOOOBJECT(self)->pos >= 0);
return PyString_FromStringAndSize(output, n);
}
@@ -277,6 +287,7 @@
if (!IO__opencheck(self)) return NULL;
+ assert(self->pos >= 0);
return PyInt_FromSsize_t(self->pos);
}
Modified: python/branches/tlee-ast-optimize/Modules/cjkcodecs/multibytecodec.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/cjkcodecs/multibytecodec.c (original)
+++ python/branches/tlee-ast-optimize/Modules/cjkcodecs/multibytecodec.c Fri Jun 13 16:43:12 2008
@@ -163,13 +163,17 @@
static int
expand_encodebuffer(MultibyteEncodeBuffer *buf, Py_ssize_t esize)
{
- Py_ssize_t orgpos, orgsize;
+ Py_ssize_t orgpos, orgsize, incsize;
orgpos = (Py_ssize_t)((char *)buf->outbuf -
PyString_AS_STRING(buf->outobj));
orgsize = PyString_GET_SIZE(buf->outobj);
- if (_PyString_Resize(&buf->outobj, orgsize + (
- esize < (orgsize >> 1) ? (orgsize >> 1) | 1 : esize)) == -1)
+ incsize = (esize < (orgsize >> 1) ? (orgsize >> 1) | 1 : esize);
+
+ if (orgsize > PY_SSIZE_T_MAX - incsize)
+ return -1;
+
+ if (_PyString_Resize(&buf->outobj, orgsize + incsize) == -1)
return -1;
buf->outbuf = (unsigned char *)PyString_AS_STRING(buf->outobj) +orgpos;
@@ -473,6 +477,12 @@
buf.excobj = NULL;
buf.inbuf = buf.inbuf_top = *data;
buf.inbuf_end = buf.inbuf_top + datalen;
+
+ if (datalen > (PY_SSIZE_T_MAX - 16) / 2) {
+ PyErr_NoMemory();
+ goto errorexit;
+ }
+
buf.outobj = PyString_FromStringAndSize(NULL, datalen * 2 + 16);
if (buf.outobj == NULL)
goto errorexit;
@@ -735,6 +745,11 @@
origpending = ctx->pendingsize;
if (origpending > 0) {
+ if (datalen > PY_SSIZE_T_MAX - ctx->pendingsize) {
+ PyErr_NoMemory();
+ /* inbuf_tmp == NULL */
+ goto errorexit;
+ }
inbuf_tmp = PyMem_New(Py_UNICODE, datalen + ctx->pendingsize);
if (inbuf_tmp == NULL)
goto errorexit;
@@ -797,9 +812,10 @@
Py_ssize_t npendings;
npendings = (Py_ssize_t)(buf->inbuf_end - buf->inbuf);
- if (npendings + ctx->pendingsize > MAXDECPENDING) {
- PyErr_SetString(PyExc_UnicodeError, "pending buffer overflow");
- return -1;
+ if (npendings + ctx->pendingsize > MAXDECPENDING ||
+ npendings > PY_SSIZE_T_MAX - ctx->pendingsize) {
+ PyErr_SetString(PyExc_UnicodeError, "pending buffer overflow");
+ return -1;
}
memcpy(ctx->pending + ctx->pendingsize, buf->inbuf, npendings);
ctx->pendingsize += npendings;
@@ -1001,7 +1017,7 @@
PyObject *args, PyObject *kwargs)
{
MultibyteDecodeBuffer buf;
- char *data, *wdata;
+ char *data, *wdata = NULL;
Py_ssize_t wsize, finalsize = 0, size, origpending;
int final = 0;
@@ -1017,6 +1033,10 @@
wdata = data;
}
else {
+ if (size > PY_SSIZE_T_MAX - self->pendingsize) {
+ PyErr_NoMemory();
+ goto errorexit;
+ }
wsize = size + self->pendingsize;
wdata = PyMem_Malloc(wsize);
if (wdata == NULL)
@@ -1235,6 +1255,10 @@
PyObject *ctr;
char *ctrdata;
+ if (PyString_GET_SIZE(cres) > PY_SSIZE_T_MAX - self->pendingsize) {
+ PyErr_NoMemory();
+ goto errorexit;
+ }
rsize = PyString_GET_SIZE(cres) + self->pendingsize;
ctr = PyString_FromStringAndSize(NULL, rsize);
if (ctr == NULL)
Modified: python/branches/tlee-ast-optimize/Modules/datetimemodule.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/datetimemodule.c (original)
+++ python/branches/tlee-ast-optimize/Modules/datetimemodule.c Fri Jun 13 16:43:12 2008
@@ -1115,6 +1115,8 @@
char sign;
int none;
+ assert(buflen >= 1);
+
offset = call_utcoffset(tzinfo, tzinfoarg, &none);
if (offset == -1 && PyErr_Occurred())
return -1;
@@ -1206,10 +1208,15 @@
* a new format. Since computing the replacements for those codes
* is expensive, don't unless they're actually used.
*/
+ if (format_len > INT_MAX - 1) {
+ PyErr_NoMemory();
+ goto Done;
+ }
+
totalnew = format_len + 1; /* realistic if no %z/%Z/%f */
- newfmt = PyBytes_FromStringAndSize(NULL, totalnew);
+ newfmt = PyString_FromStringAndSize(NULL, totalnew);
if (newfmt == NULL) goto Done;
- pnew = PyBytes_AsString(newfmt);
+ pnew = PyString_AsString(newfmt);
usednew = 0;
pin = format;
Modified: python/branches/tlee-ast-optimize/Modules/md5.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/md5.c (original)
+++ python/branches/tlee-ast-optimize/Modules/md5.c Fri Jun 13 16:43:12 2008
@@ -53,6 +53,7 @@
#include "md5.h"
#include <string.h>
+#include <limits.h>
#undef BYTE_ORDER /* 1 = big-endian, -1 = little-endian, 0 = unknown */
#ifdef ARCH_IS_BIG_ENDIAN
@@ -330,6 +331,18 @@
if (nbytes <= 0)
return;
+ /* this special case is handled recursively */
+ if (nbytes > INT_MAX - offset) {
+ int overlap;
+
+ /* handle the append in two steps to prevent overflow */
+ overlap = 64 - offset;
+
+ md5_append(pms, data, overlap);
+ md5_append(pms, data + overlap, nbytes - overlap);
+ return;
+ }
+
/* Update the message length. */
pms->count[1] += nbytes >> 29;
pms->count[0] += nbits;
Modified: python/branches/tlee-ast-optimize/Modules/stropmodule.c
==============================================================================
--- python/branches/tlee-ast-optimize/Modules/stropmodule.c (original)
+++ python/branches/tlee-ast-optimize/Modules/stropmodule.c Fri Jun 13 16:43:12 2008
@@ -578,7 +578,7 @@
char* e;
char* p;
char* q;
- Py_ssize_t i, j;
+ Py_ssize_t i, j, old_j;
PyObject* out;
char* string;
Py_ssize_t stringlen;
@@ -595,12 +595,18 @@
}
/* First pass: determine size of output string */
- i = j = 0; /* j: current column; i: total of previous lines */
+ i = j = old_j = 0; /* j: current column; i: total of previous lines */
e = string + stringlen;
for (p = string; p < e; p++) {
- if (*p == '\t')
+ if (*p == '\t') {
j += tabsize - (j%tabsize);
- else {
+ if (old_j > j) {
+ PyErr_SetString(PyExc_OverflowError,
+ "new string is too long");
+ return NULL;
+ }
+ old_j = j;
+ } else {
j++;
if (*p == '\n') {
i += j;
@@ -609,6 +615,11 @@
}
}
+ if ((i + j) < 0) {
+ PyErr_SetString(PyExc_OverflowError, "new string is too long");
+ return NULL;
+ }
+
/* Second pass: create output string and fill it */
out = PyString_FromStringAndSize(NULL, i+j);
if (out == NULL)
Modified: python/branches/tlee-ast-optimize/Objects/bufferobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/bufferobject.c (original)
+++ python/branches/tlee-ast-optimize/Objects/bufferobject.c Fri Jun 13 16:43:12 2008
@@ -207,7 +207,10 @@
"size must be zero or positive");
return NULL;
}
- /* XXX: check for overflow in multiply */
+ if (sizeof(*b) > PY_SSIZE_T_MAX - size) {
+ /* unlikely */
+ return PyErr_NoMemory();
+ }
/* Inline PyObject_New */
o = (PyObject *)PyObject_MALLOC(sizeof(*b) + size);
if ( o == NULL )
@@ -401,6 +404,8 @@
if ( (count = (*pb->bf_getreadbuffer)(other, 0, &ptr2)) < 0 )
return NULL;
+ assert(count <= PY_SIZE_MAX - size);
+
ob = PyString_FromStringAndSize(NULL, size + count);
if ( ob == NULL )
return NULL;
Deleted: python/branches/tlee-ast-optimize/Objects/bytesobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/bytesobject.c Fri Jun 13 16:43:12 2008
+++ (empty file)
@@ -1,5215 +0,0 @@
-/* String object implementation */
-
-#define PY_SSIZE_T_CLEAN
-
-#include "Python.h"
-#include <ctype.h>
-
-#ifdef COUNT_ALLOCS
-int null_strings, one_strings;
-#endif
-
-static PyBytesObject *characters[UCHAR_MAX + 1];
-static PyBytesObject *nullstring;
-
-/* This dictionary holds all interned strings. Note that references to
- strings in this dictionary are *not* counted in the string's ob_refcnt.
- When the interned string reaches a refcnt of 0 the string deallocation
- function will delete the reference from this dictionary.
-
- Another way to look at this is that to say that the actual reference
- count of a string is: s->ob_refcnt + (s->ob_sstate?2:0)
-*/
-static PyObject *interned;
-
-/*
- For both PyBytes_FromString() and PyBytes_FromStringAndSize(), the
- parameter `size' denotes number of characters to allocate, not counting any
- null terminating character.
-
- For PyBytes_FromString(), the parameter `str' points to a null-terminated
- string containing exactly `size' bytes.
-
- For PyBytes_FromStringAndSize(), the parameter the parameter `str' is
- either NULL or else points to a string containing at least `size' bytes.
- For PyBytes_FromStringAndSize(), the string in the `str' parameter does
- not have to be null-terminated. (Therefore it is safe to construct a
- substring by calling `PyBytes_FromStringAndSize(origstring, substrlen)'.)
- If `str' is NULL then PyBytes_FromStringAndSize() will allocate `size+1'
- bytes (setting the last byte to the null terminating character) and you can
- fill in the data yourself. If `str' is non-NULL then the resulting
- PyString object must be treated as immutable and you must not fill in nor
- alter the data yourself, since the strings may be shared.
-
- The PyObject member `op->ob_size', which denotes the number of "extra
- items" in a variable-size object, will contain the number of bytes
- allocated for string data, not counting the null terminating character. It
- is therefore equal to the equal to the `size' parameter (for
- PyBytes_FromStringAndSize()) or the length of the string in the `str'
- parameter (for PyBytes_FromString()).
-*/
-PyObject *
-PyBytes_FromStringAndSize(const char *str, Py_ssize_t size)
-{
- register PyBytesObject *op;
- if (size < 0) {
- PyErr_SetString(PyExc_SystemError,
- "Negative size passed to PyBytes_FromStringAndSize");
- return NULL;
- }
- if (size == 0 && (op = nullstring) != NULL) {
-#ifdef COUNT_ALLOCS
- null_strings++;
-#endif
- Py_INCREF(op);
- return (PyObject *)op;
- }
- if (size == 1 && str != NULL &&
- (op = characters[*str & UCHAR_MAX]) != NULL)
- {
-#ifdef COUNT_ALLOCS
- one_strings++;
-#endif
- Py_INCREF(op);
- return (PyObject *)op;
- }
-
- /* Inline PyObject_NewVar */
- op = (PyBytesObject *)PyObject_MALLOC(sizeof(PyBytesObject) + size);
- if (op == NULL)
- return PyErr_NoMemory();
- PyObject_INIT_VAR(op, &PyBytes_Type, size);
- op->ob_shash = -1;
- op->ob_sstate = SSTATE_NOT_INTERNED;
- if (str != NULL)
- Py_MEMCPY(op->ob_sval, str, size);
- op->ob_sval[size] = '\0';
- /* share short strings */
- if (size == 0) {
- PyObject *t = (PyObject *)op;
- PyString_InternInPlace(&t);
- op = (PyBytesObject *)t;
- nullstring = op;
- Py_INCREF(op);
- } else if (size == 1 && str != NULL) {
- PyObject *t = (PyObject *)op;
- PyString_InternInPlace(&t);
- op = (PyBytesObject *)t;
- characters[*str & UCHAR_MAX] = op;
- Py_INCREF(op);
- }
- return (PyObject *) op;
-}
-
-PyObject *
-PyBytes_FromString(const char *str)
-{
- register size_t size;
- register PyBytesObject *op;
-
- assert(str != NULL);
- size = strlen(str);
- if (size > PY_SSIZE_T_MAX) {
- PyErr_SetString(PyExc_OverflowError,
- "string is too long for a Python string");
- return NULL;
- }
- if (size == 0 && (op = nullstring) != NULL) {
-#ifdef COUNT_ALLOCS
- null_strings++;
-#endif
- Py_INCREF(op);
- return (PyObject *)op;
- }
- if (size == 1 && (op = characters[*str & UCHAR_MAX]) != NULL) {
-#ifdef COUNT_ALLOCS
- one_strings++;
-#endif
- Py_INCREF(op);
- return (PyObject *)op;
- }
-
- /* Inline PyObject_NewVar */
- op = (PyBytesObject *)PyObject_MALLOC(sizeof(PyBytesObject) + size);
- if (op == NULL)
- return PyErr_NoMemory();
- PyObject_INIT_VAR(op, &PyBytes_Type, size);
- op->ob_shash = -1;
- op->ob_sstate = SSTATE_NOT_INTERNED;
- Py_MEMCPY(op->ob_sval, str, size+1);
- /* share short strings */
- if (size == 0) {
- PyObject *t = (PyObject *)op;
- PyString_InternInPlace(&t);
- op = (PyBytesObject *)t;
- nullstring = op;
- Py_INCREF(op);
- } else if (size == 1) {
- PyObject *t = (PyObject *)op;
- PyString_InternInPlace(&t);
- op = (PyBytesObject *)t;
- characters[*str & UCHAR_MAX] = op;
- Py_INCREF(op);
- }
- return (PyObject *) op;
-}
-
-PyObject *
-PyBytes_FromFormatV(const char *format, va_list vargs)
-{
- va_list count;
- Py_ssize_t n = 0;
- const char* f;
- char *s;
- PyObject* string;
-
-#ifdef VA_LIST_IS_ARRAY
- Py_MEMCPY(count, vargs, sizeof(va_list));
-#else
-#ifdef __va_copy
- __va_copy(count, vargs);
-#else
- count = vargs;
-#endif
-#endif
- /* step 1: figure out how large a buffer we need */
- for (f = format; *f; f++) {
- if (*f == '%') {
- const char* p = f;
- while (*++f && *f != '%' && !isalpha(Py_CHARMASK(*f)))
- ;
-
- /* skip the 'l' or 'z' in {%ld, %zd, %lu, %zu} since
- * they don't affect the amount of space we reserve.
- */
- if ((*f == 'l' || *f == 'z') &&
- (f[1] == 'd' || f[1] == 'u'))
- ++f;
-
- switch (*f) {
- case 'c':
- (void)va_arg(count, int);
- /* fall through... */
- case '%':
- n++;
- break;
- case 'd': case 'u': case 'i': case 'x':
- (void) va_arg(count, int);
- /* 20 bytes is enough to hold a 64-bit
- integer. Decimal takes the most space.
- This isn't enough for octal. */
- n += 20;
- break;
- case 's':
- s = va_arg(count, char*);
- n += strlen(s);
- break;
- case 'p':
- (void) va_arg(count, int);
- /* maximum 64-bit pointer representation:
- * 0xffffffffffffffff
- * so 19 characters is enough.
- * XXX I count 18 -- what's the extra for?
- */
- n += 19;
- break;
- default:
- /* if we stumble upon an unknown
- formatting code, copy the rest of
- the format string to the output
- string. (we cannot just skip the
- code, since there's no way to know
- what's in the argument list) */
- n += strlen(p);
- goto expand;
- }
- } else
- n++;
- }
- expand:
- /* step 2: fill the buffer */
- /* Since we've analyzed how much space we need for the worst case,
- use sprintf directly instead of the slower PyOS_snprintf. */
- string = PyBytes_FromStringAndSize(NULL, n);
- if (!string)
- return NULL;
-
- s = PyBytes_AsString(string);
-
- for (f = format; *f; f++) {
- if (*f == '%') {
- const char* p = f++;
- Py_ssize_t i;
- int longflag = 0;
- int size_tflag = 0;
- /* parse the width.precision part (we're only
- interested in the precision value, if any) */
- n = 0;
- while (isdigit(Py_CHARMASK(*f)))
- n = (n*10) + *f++ - '0';
- if (*f == '.') {
- f++;
- n = 0;
- while (isdigit(Py_CHARMASK(*f)))
- n = (n*10) + *f++ - '0';
- }
- while (*f && *f != '%' && !isalpha(Py_CHARMASK(*f)))
- f++;
- /* handle the long flag, but only for %ld and %lu.
- others can be added when necessary. */
- if (*f == 'l' && (f[1] == 'd' || f[1] == 'u')) {
- longflag = 1;
- ++f;
- }
- /* handle the size_t flag. */
- if (*f == 'z' && (f[1] == 'd' || f[1] == 'u')) {
- size_tflag = 1;
- ++f;
- }
-
- switch (*f) {
- case 'c':
- *s++ = va_arg(vargs, int);
- break;
- case 'd':
- if (longflag)
- sprintf(s, "%ld", va_arg(vargs, long));
- else if (size_tflag)
- sprintf(s, "%" PY_FORMAT_SIZE_T "d",
- va_arg(vargs, Py_ssize_t));
- else
- sprintf(s, "%d", va_arg(vargs, int));
- s += strlen(s);
- break;
- case 'u':
- if (longflag)
- sprintf(s, "%lu",
- va_arg(vargs, unsigned long));
- else if (size_tflag)
- sprintf(s, "%" PY_FORMAT_SIZE_T "u",
- va_arg(vargs, size_t));
- else
- sprintf(s, "%u",
- va_arg(vargs, unsigned int));
- s += strlen(s);
- break;
- case 'i':
- sprintf(s, "%i", va_arg(vargs, int));
- s += strlen(s);
- break;
- case 'x':
- sprintf(s, "%x", va_arg(vargs, int));
- s += strlen(s);
- break;
- case 's':
- p = va_arg(vargs, char*);
- i = strlen(p);
- if (n > 0 && i > n)
- i = n;
- Py_MEMCPY(s, p, i);
- s += i;
- break;
- case 'p':
- sprintf(s, "%p", va_arg(vargs, void*));
- /* %p is ill-defined: ensure leading 0x. */
- if (s[1] == 'X')
- s[1] = 'x';
- else if (s[1] != 'x') {
- memmove(s+2, s, strlen(s)+1);
- s[0] = '0';
- s[1] = 'x';
- }
- s += strlen(s);
- break;
- case '%':
- *s++ = '%';
- break;
- default:
- strcpy(s, p);
- s += strlen(s);
- goto end;
- }
- } else
- *s++ = *f;
- }
-
- end:
- _PyBytes_Resize(&string, s - PyBytes_AS_STRING(string));
- return string;
-}
-
-PyObject *
-PyBytes_FromFormat(const char *format, ...)
-{
- PyObject* ret;
- va_list vargs;
-
-#ifdef HAVE_STDARG_PROTOTYPES
- va_start(vargs, format);
-#else
- va_start(vargs);
-#endif
- ret = PyBytes_FromFormatV(format, vargs);
- va_end(vargs);
- return ret;
-}
-
-
-PyObject *PyBytes_Decode(const char *s,
- Py_ssize_t size,
- const char *encoding,
- const char *errors)
-{
- PyObject *v, *str;
-
- str = PyBytes_FromStringAndSize(s, size);
- if (str == NULL)
- return NULL;
- v = PyBytes_AsDecodedString(str, encoding, errors);
- Py_DECREF(str);
- return v;
-}
-
-PyObject *PyBytes_AsDecodedObject(PyObject *str,
- const char *encoding,
- const char *errors)
-{
- PyObject *v;
-
- if (!PyBytes_Check(str)) {
- PyErr_BadArgument();
- goto onError;
- }
-
- if (encoding == NULL) {
-#ifdef Py_USING_UNICODE
- encoding = PyUnicode_GetDefaultEncoding();
-#else
- PyErr_SetString(PyExc_ValueError, "no encoding specified");
- goto onError;
-#endif
- }
-
- /* Decode via the codec registry */
- v = PyCodec_Decode(str, encoding, errors);
- if (v == NULL)
- goto onError;
-
- return v;
-
- onError:
- return NULL;
-}
-
-PyObject *PyBytes_AsDecodedString(PyObject *str,
- const char *encoding,
- const char *errors)
-{
- PyObject *v;
-
- v = PyBytes_AsDecodedObject(str, encoding, errors);
- if (v == NULL)
- goto onError;
-
-#ifdef Py_USING_UNICODE
- /* Convert Unicode to a string using the default encoding */
- if (PyUnicode_Check(v)) {
- PyObject *temp = v;
- v = PyUnicode_AsEncodedString(v, NULL, NULL);
- Py_DECREF(temp);
- if (v == NULL)
- goto onError;
- }
-#endif
- if (!PyBytes_Check(v)) {
- PyErr_Format(PyExc_TypeError,
- "decoder did not return a string object (type=%.400s)",
- Py_TYPE(v)->tp_name);
- Py_DECREF(v);
- goto onError;
- }
-
- return v;
-
- onError:
- return NULL;
-}
-
-PyObject *PyBytes_Encode(const char *s,
- Py_ssize_t size,
- const char *encoding,
- const char *errors)
-{
- PyObject *v, *str;
-
- str = PyBytes_FromStringAndSize(s, size);
- if (str == NULL)
- return NULL;
- v = PyBytes_AsEncodedString(str, encoding, errors);
- Py_DECREF(str);
- return v;
-}
-
-PyObject *PyBytes_AsEncodedObject(PyObject *str,
- const char *encoding,
- const char *errors)
-{
- PyObject *v;
-
- if (!PyBytes_Check(str)) {
- PyErr_BadArgument();
- goto onError;
- }
-
- if (encoding == NULL) {
-#ifdef Py_USING_UNICODE
- encoding = PyUnicode_GetDefaultEncoding();
-#else
- PyErr_SetString(PyExc_ValueError, "no encoding specified");
- goto onError;
-#endif
- }
-
- /* Encode via the codec registry */
- v = PyCodec_Encode(str, encoding, errors);
- if (v == NULL)
- goto onError;
-
- return v;
-
- onError:
- return NULL;
-}
-
-PyObject *PyBytes_AsEncodedString(PyObject *str,
- const char *encoding,
- const char *errors)
-{
- PyObject *v;
-
- v = PyBytes_AsEncodedObject(str, encoding, errors);
- if (v == NULL)
- goto onError;
-
-#ifdef Py_USING_UNICODE
- /* Convert Unicode to a string using the default encoding */
- if (PyUnicode_Check(v)) {
- PyObject *temp = v;
- v = PyUnicode_AsEncodedString(v, NULL, NULL);
- Py_DECREF(temp);
- if (v == NULL)
- goto onError;
- }
-#endif
- if (!PyBytes_Check(v)) {
- PyErr_Format(PyExc_TypeError,
- "encoder did not return a string object (type=%.400s)",
- Py_TYPE(v)->tp_name);
- Py_DECREF(v);
- goto onError;
- }
-
- return v;
-
- onError:
- return NULL;
-}
-
-static void
-string_dealloc(PyObject *op)
-{
- switch (PyBytes_CHECK_INTERNED(op)) {
- case SSTATE_NOT_INTERNED:
- break;
-
- case SSTATE_INTERNED_MORTAL:
- /* revive dead object temporarily for DelItem */
- Py_REFCNT(op) = 3;
- if (PyDict_DelItem(interned, op) != 0)
- Py_FatalError(
- "deletion of interned string failed");
- break;
-
- case SSTATE_INTERNED_IMMORTAL:
- Py_FatalError("Immortal interned string died.");
-
- default:
- Py_FatalError("Inconsistent interned string state.");
- }
- Py_TYPE(op)->tp_free(op);
-}
-
-/* Unescape a backslash-escaped string. If unicode is non-zero,
- the string is a u-literal. If recode_encoding is non-zero,
- the string is UTF-8 encoded and should be re-encoded in the
- specified encoding. */
-
-PyObject *PyBytes_DecodeEscape(const char *s,
- Py_ssize_t len,
- const char *errors,
- Py_ssize_t unicode,
- const char *recode_encoding)
-{
- int c;
- char *p, *buf;
- const char *end;
- PyObject *v;
- Py_ssize_t newlen = recode_encoding ? 4*len:len;
- v = PyBytes_FromStringAndSize((char *)NULL, newlen);
- if (v == NULL)
- return NULL;
- p = buf = PyBytes_AsString(v);
- end = s + len;
- while (s < end) {
- if (*s != '\\') {
- non_esc:
-#ifdef Py_USING_UNICODE
- if (recode_encoding && (*s & 0x80)) {
- PyObject *u, *w;
- char *r;
- const char* t;
- Py_ssize_t rn;
- t = s;
- /* Decode non-ASCII bytes as UTF-8. */
- while (t < end && (*t & 0x80)) t++;
- u = PyUnicode_DecodeUTF8(s, t - s, errors);
- if(!u) goto failed;
-
- /* Recode them in target encoding. */
- w = PyUnicode_AsEncodedString(
- u, recode_encoding, errors);
- Py_DECREF(u);
- if (!w) goto failed;
-
- /* Append bytes to output buffer. */
- assert(PyBytes_Check(w));
- r = PyBytes_AS_STRING(w);
- rn = PyBytes_GET_SIZE(w);
- Py_MEMCPY(p, r, rn);
- p += rn;
- Py_DECREF(w);
- s = t;
- } else {
- *p++ = *s++;
- }
-#else
- *p++ = *s++;
-#endif
- continue;
- }
- s++;
- if (s==end) {
- PyErr_SetString(PyExc_ValueError,
- "Trailing \\ in string");
- goto failed;
- }
- switch (*s++) {
- /* XXX This assumes ASCII! */
- case '\n': break;
- case '\\': *p++ = '\\'; break;
- case '\'': *p++ = '\''; break;
- case '\"': *p++ = '\"'; break;
- case 'b': *p++ = '\b'; break;
- case 'f': *p++ = '\014'; break; /* FF */
- case 't': *p++ = '\t'; break;
- case 'n': *p++ = '\n'; break;
- case 'r': *p++ = '\r'; break;
- case 'v': *p++ = '\013'; break; /* VT */
- case 'a': *p++ = '\007'; break; /* BEL, not classic C */
- case '0': case '1': case '2': case '3':
- case '4': case '5': case '6': case '7':
- c = s[-1] - '0';
- if (s < end && '0' <= *s && *s <= '7') {
- c = (c<<3) + *s++ - '0';
- if (s < end && '0' <= *s && *s <= '7')
- c = (c<<3) + *s++ - '0';
- }
- *p++ = c;
- break;
- case 'x':
- if (s+1 < end &&
- isxdigit(Py_CHARMASK(s[0])) &&
- isxdigit(Py_CHARMASK(s[1])))
- {
- unsigned int x = 0;
- c = Py_CHARMASK(*s);
- s++;
- if (isdigit(c))
- x = c - '0';
- else if (islower(c))
- x = 10 + c - 'a';
- else
- x = 10 + c - 'A';
- x = x << 4;
- c = Py_CHARMASK(*s);
- s++;
- if (isdigit(c))
- x += c - '0';
- else if (islower(c))
- x += 10 + c - 'a';
- else
- x += 10 + c - 'A';
- *p++ = x;
- break;
- }
- if (!errors || strcmp(errors, "strict") == 0) {
- PyErr_SetString(PyExc_ValueError,
- "invalid \\x escape");
- goto failed;
- }
- if (strcmp(errors, "replace") == 0) {
- *p++ = '?';
- } else if (strcmp(errors, "ignore") == 0)
- /* do nothing */;
- else {
- PyErr_Format(PyExc_ValueError,
- "decoding error; "
- "unknown error handling code: %.400s",
- errors);
- goto failed;
- }
-#ifndef Py_USING_UNICODE
- case 'u':
- case 'U':
- case 'N':
- if (unicode) {
- PyErr_SetString(PyExc_ValueError,
- "Unicode escapes not legal "
- "when Unicode disabled");
- goto failed;
- }
-#endif
- default:
- *p++ = '\\';
- s--;
- goto non_esc; /* an arbitry number of unescaped
- UTF-8 bytes may follow. */
- }
- }
- if (p-buf < newlen)
- _PyBytes_Resize(&v, p - buf);
- return v;
- failed:
- Py_DECREF(v);
- return NULL;
-}
-
-/* -------------------------------------------------------------------- */
-/* object api */
-
-static Py_ssize_t
-string_getsize(register PyObject *op)
-{
- char *s;
- Py_ssize_t len;
- if (PyBytes_AsStringAndSize(op, &s, &len))
- return -1;
- return len;
-}
-
-static /*const*/ char *
-string_getbuffer(register PyObject *op)
-{
- char *s;
- Py_ssize_t len;
- if (PyBytes_AsStringAndSize(op, &s, &len))
- return NULL;
- return s;
-}
-
-Py_ssize_t
-PyBytes_Size(register PyObject *op)
-{
- if (!PyBytes_Check(op))
- return string_getsize(op);
- return Py_SIZE(op);
-}
-
-/*const*/ char *
-PyBytes_AsString(register PyObject *op)
-{
- if (!PyBytes_Check(op))
- return string_getbuffer(op);
- return ((PyBytesObject *)op) -> ob_sval;
-}
-
-int
-PyBytes_AsStringAndSize(register PyObject *obj,
- register char **s,
- register Py_ssize_t *len)
-{
- if (s == NULL) {
- PyErr_BadInternalCall();
- return -1;
- }
-
- if (!PyBytes_Check(obj)) {
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(obj)) {
- obj = _PyUnicode_AsDefaultEncodedString(obj, NULL);
- if (obj == NULL)
- return -1;
- }
- else
-#endif
- {
- PyErr_Format(PyExc_TypeError,
- "expected string or Unicode object, "
- "%.200s found", Py_TYPE(obj)->tp_name);
- return -1;
- }
- }
-
- *s = PyBytes_AS_STRING(obj);
- if (len != NULL)
- *len = PyBytes_GET_SIZE(obj);
- else if (strlen(*s) != (size_t)PyBytes_GET_SIZE(obj)) {
- PyErr_SetString(PyExc_TypeError,
- "expected string without null bytes");
- return -1;
- }
- return 0;
-}
-
-/* -------------------------------------------------------------------- */
-/* Methods */
-
-#include "stringlib/stringdefs.h"
-#include "stringlib/fastsearch.h"
-
-#include "stringlib/count.h"
-#include "stringlib/find.h"
-#include "stringlib/partition.h"
-
-#define _Py_InsertThousandsGrouping _PyBytes_InsertThousandsGrouping
-#include "stringlib/localeutil.h"
-
-
-
-static int
-string_print(PyBytesObject *op, FILE *fp, int flags)
-{
- Py_ssize_t i, str_len;
- char c;
- int quote;
-
- /* XXX Ought to check for interrupts when writing long strings */
- if (! PyBytes_CheckExact(op)) {
- int ret;
- /* A str subclass may have its own __str__ method. */
- op = (PyBytesObject *) PyObject_Str((PyObject *)op);
- if (op == NULL)
- return -1;
- ret = string_print(op, fp, flags);
- Py_DECREF(op);
- return ret;
- }
- if (flags & Py_PRINT_RAW) {
- char *data = op->ob_sval;
- Py_ssize_t size = Py_SIZE(op);
- Py_BEGIN_ALLOW_THREADS
- while (size > INT_MAX) {
- /* Very long strings cannot be written atomically.
- * But don't write exactly INT_MAX bytes at a time
- * to avoid memory aligment issues.
- */
- const int chunk_size = INT_MAX & ~0x3FFF;
- fwrite(data, 1, chunk_size, fp);
- data += chunk_size;
- size -= chunk_size;
- }
-#ifdef __VMS
- if (size) fwrite(data, (int)size, 1, fp);
-#else
- fwrite(data, 1, (int)size, fp);
-#endif
- Py_END_ALLOW_THREADS
- return 0;
- }
-
- /* figure out which quote to use; single is preferred */
- quote = '\'';
- if (memchr(op->ob_sval, '\'', Py_SIZE(op)) &&
- !memchr(op->ob_sval, '"', Py_SIZE(op)))
- quote = '"';
-
- str_len = Py_SIZE(op);
- Py_BEGIN_ALLOW_THREADS
- fputc(quote, fp);
- for (i = 0; i < str_len; i++) {
- /* Since strings are immutable and the caller should have a
- reference, accessing the interal buffer should not be an issue
- with the GIL released. */
- c = op->ob_sval[i];
- if (c == quote || c == '\\')
- fprintf(fp, "\\%c", c);
- else if (c == '\t')
- fprintf(fp, "\\t");
- else if (c == '\n')
- fprintf(fp, "\\n");
- else if (c == '\r')
- fprintf(fp, "\\r");
- else if (c < ' ' || c >= 0x7f)
- fprintf(fp, "\\x%02x", c & 0xff);
- else
- fputc(c, fp);
- }
- fputc(quote, fp);
- Py_END_ALLOW_THREADS
- return 0;
-}
-
-PyObject *
-PyBytes_Repr(PyObject *obj, int smartquotes)
-{
- register PyBytesObject* op = (PyBytesObject*) obj;
- size_t newsize = 2 + 4 * Py_SIZE(op);
- PyObject *v;
- if (newsize > PY_SSIZE_T_MAX || newsize / 4 != Py_SIZE(op)) {
- PyErr_SetString(PyExc_OverflowError,
- "string is too large to make repr");
- return NULL;
- }
- v = PyBytes_FromStringAndSize((char *)NULL, newsize);
- if (v == NULL) {
- return NULL;
- }
- else {
- register Py_ssize_t i;
- register char c;
- register char *p;
- int quote;
-
- /* figure out which quote to use; single is preferred */
- quote = '\'';
- if (smartquotes &&
- memchr(op->ob_sval, '\'', Py_SIZE(op)) &&
- !memchr(op->ob_sval, '"', Py_SIZE(op)))
- quote = '"';
-
- p = PyBytes_AS_STRING(v);
- *p++ = quote;
- for (i = 0; i < Py_SIZE(op); i++) {
- /* There's at least enough room for a hex escape
- and a closing quote. */
- assert(newsize - (p - PyBytes_AS_STRING(v)) >= 5);
- c = op->ob_sval[i];
- if (c == quote || c == '\\')
- *p++ = '\\', *p++ = c;
- else if (c == '\t')
- *p++ = '\\', *p++ = 't';
- else if (c == '\n')
- *p++ = '\\', *p++ = 'n';
- else if (c == '\r')
- *p++ = '\\', *p++ = 'r';
- else if (c < ' ' || c >= 0x7f) {
- /* For performance, we don't want to call
- PyOS_snprintf here (extra layers of
- function call). */
- sprintf(p, "\\x%02x", c & 0xff);
- p += 4;
- }
- else
- *p++ = c;
- }
- assert(newsize - (p - PyBytes_AS_STRING(v)) >= 1);
- *p++ = quote;
- *p = '\0';
- _PyBytes_Resize(
- &v, (p - PyBytes_AS_STRING(v)));
- return v;
- }
-}
-
-static PyObject *
-string_repr(PyObject *op)
-{
- return PyBytes_Repr(op, 1);
-}
-
-static PyObject *
-string_str(PyObject *s)
-{
- assert(PyBytes_Check(s));
- if (PyBytes_CheckExact(s)) {
- Py_INCREF(s);
- return s;
- }
- else {
- /* Subtype -- return genuine string with the same value. */
- PyBytesObject *t = (PyBytesObject *) s;
- return PyBytes_FromStringAndSize(t->ob_sval, Py_SIZE(t));
- }
-}
-
-static Py_ssize_t
-string_length(PyBytesObject *a)
-{
- return Py_SIZE(a);
-}
-
-static PyObject *
-string_concat(register PyBytesObject *a, register PyObject *bb)
-{
- register Py_ssize_t size;
- register PyBytesObject *op;
- if (!PyBytes_Check(bb)) {
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(bb))
- return PyUnicode_Concat((PyObject *)a, bb);
-#endif
- if (PyByteArray_Check(bb))
- return PyByteArray_Concat((PyObject *)a, bb);
- PyErr_Format(PyExc_TypeError,
- "cannot concatenate 'str' and '%.200s' objects",
- Py_TYPE(bb)->tp_name);
- return NULL;
- }
-#define b ((PyBytesObject *)bb)
- /* Optimize cases with empty left or right operand */
- if ((Py_SIZE(a) == 0 || Py_SIZE(b) == 0) &&
- PyBytes_CheckExact(a) && PyBytes_CheckExact(b)) {
- if (Py_SIZE(a) == 0) {
- Py_INCREF(bb);
- return bb;
- }
- Py_INCREF(a);
- return (PyObject *)a;
- }
- size = Py_SIZE(a) + Py_SIZE(b);
- if (size < 0) {
- PyErr_SetString(PyExc_OverflowError,
- "strings are too large to concat");
- return NULL;
- }
-
- /* Inline PyObject_NewVar */
- op = (PyBytesObject *)PyObject_MALLOC(sizeof(PyBytesObject) + size);
- if (op == NULL)
- return PyErr_NoMemory();
- PyObject_INIT_VAR(op, &PyBytes_Type, size);
- op->ob_shash = -1;
- op->ob_sstate = SSTATE_NOT_INTERNED;
- Py_MEMCPY(op->ob_sval, a->ob_sval, Py_SIZE(a));
- Py_MEMCPY(op->ob_sval + Py_SIZE(a), b->ob_sval, Py_SIZE(b));
- op->ob_sval[size] = '\0';
- return (PyObject *) op;
-#undef b
-}
-
-static PyObject *
-string_repeat(register PyBytesObject *a, register Py_ssize_t n)
-{
- register Py_ssize_t i;
- register Py_ssize_t j;
- register Py_ssize_t size;
- register PyBytesObject *op;
- size_t nbytes;
- if (n < 0)
- n = 0;
- /* watch out for overflows: the size can overflow int,
- * and the # of bytes needed can overflow size_t
- */
- size = Py_SIZE(a) * n;
- if (n && size / n != Py_SIZE(a)) {
- PyErr_SetString(PyExc_OverflowError,
- "repeated string is too long");
- return NULL;
- }
- if (size == Py_SIZE(a) && PyBytes_CheckExact(a)) {
- Py_INCREF(a);
- return (PyObject *)a;
- }
- nbytes = (size_t)size;
- if (nbytes + sizeof(PyBytesObject) <= nbytes) {
- PyErr_SetString(PyExc_OverflowError,
- "repeated string is too long");
- return NULL;
- }
- op = (PyBytesObject *)
- PyObject_MALLOC(sizeof(PyBytesObject) + nbytes);
- if (op == NULL)
- return PyErr_NoMemory();
- PyObject_INIT_VAR(op, &PyBytes_Type, size);
- op->ob_shash = -1;
- op->ob_sstate = SSTATE_NOT_INTERNED;
- op->ob_sval[size] = '\0';
- if (Py_SIZE(a) == 1 && n > 0) {
- memset(op->ob_sval, a->ob_sval[0] , n);
- return (PyObject *) op;
- }
- i = 0;
- if (i < size) {
- Py_MEMCPY(op->ob_sval, a->ob_sval, Py_SIZE(a));
- i = Py_SIZE(a);
- }
- while (i < size) {
- j = (i <= size-i) ? i : size-i;
- Py_MEMCPY(op->ob_sval+i, op->ob_sval, j);
- i += j;
- }
- return (PyObject *) op;
-}
-
-/* String slice a[i:j] consists of characters a[i] ... a[j-1] */
-
-static PyObject *
-string_slice(register PyBytesObject *a, register Py_ssize_t i,
- register Py_ssize_t j)
- /* j -- may be negative! */
-{
- if (i < 0)
- i = 0;
- if (j < 0)
- j = 0; /* Avoid signed/unsigned bug in next line */
- if (j > Py_SIZE(a))
- j = Py_SIZE(a);
- if (i == 0 && j == Py_SIZE(a) && PyBytes_CheckExact(a)) {
- /* It's the same as a */
- Py_INCREF(a);
- return (PyObject *)a;
- }
- if (j < i)
- j = i;
- return PyBytes_FromStringAndSize(a->ob_sval + i, j-i);
-}
-
-static int
-string_contains(PyObject *str_obj, PyObject *sub_obj)
-{
- if (!PyBytes_CheckExact(sub_obj)) {
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(sub_obj))
- return PyUnicode_Contains(str_obj, sub_obj);
-#endif
- if (!PyBytes_Check(sub_obj)) {
- PyErr_Format(PyExc_TypeError,
- "'in <string>' requires string as left operand, "
- "not %.200s", Py_TYPE(sub_obj)->tp_name);
- return -1;
- }
- }
-
- return stringlib_contains_obj(str_obj, sub_obj);
-}
-
-static PyObject *
-string_item(PyBytesObject *a, register Py_ssize_t i)
-{
- char pchar;
- PyObject *v;
- if (i < 0 || i >= Py_SIZE(a)) {
- PyErr_SetString(PyExc_IndexError, "string index out of range");
- return NULL;
- }
- pchar = a->ob_sval[i];
- v = (PyObject *)characters[pchar & UCHAR_MAX];
- if (v == NULL)
- v = PyBytes_FromStringAndSize(&pchar, 1);
- else {
-#ifdef COUNT_ALLOCS
- one_strings++;
-#endif
- Py_INCREF(v);
- }
- return v;
-}
-
-static PyObject*
-string_richcompare(PyBytesObject *a, PyBytesObject *b, int op)
-{
- int c;
- Py_ssize_t len_a, len_b;
- Py_ssize_t min_len;
- PyObject *result;
-
- /* Make sure both arguments are strings. */
- if (!(PyBytes_Check(a) && PyBytes_Check(b))) {
- result = Py_NotImplemented;
- goto out;
- }
- if (a == b) {
- switch (op) {
- case Py_EQ:case Py_LE:case Py_GE:
- result = Py_True;
- goto out;
- case Py_NE:case Py_LT:case Py_GT:
- result = Py_False;
- goto out;
- }
- }
- if (op == Py_EQ) {
- /* Supporting Py_NE here as well does not save
- much time, since Py_NE is rarely used. */
- if (Py_SIZE(a) == Py_SIZE(b)
- && (a->ob_sval[0] == b->ob_sval[0]
- && memcmp(a->ob_sval, b->ob_sval, Py_SIZE(a)) == 0)) {
- result = Py_True;
- } else {
- result = Py_False;
- }
- goto out;
- }
- len_a = Py_SIZE(a); len_b = Py_SIZE(b);
- min_len = (len_a < len_b) ? len_a : len_b;
- if (min_len > 0) {
- c = Py_CHARMASK(*a->ob_sval) - Py_CHARMASK(*b->ob_sval);
- if (c==0)
- c = memcmp(a->ob_sval, b->ob_sval, min_len);
- } else
- c = 0;
- if (c == 0)
- c = (len_a < len_b) ? -1 : (len_a > len_b) ? 1 : 0;
- switch (op) {
- case Py_LT: c = c < 0; break;
- case Py_LE: c = c <= 0; break;
- case Py_EQ: assert(0); break; /* unreachable */
- case Py_NE: c = c != 0; break;
- case Py_GT: c = c > 0; break;
- case Py_GE: c = c >= 0; break;
- default:
- result = Py_NotImplemented;
- goto out;
- }
- result = c ? Py_True : Py_False;
- out:
- Py_INCREF(result);
- return result;
-}
-
-int
-_PyBytes_Eq(PyObject *o1, PyObject *o2)
-{
- PyBytesObject *a = (PyBytesObject*) o1;
- PyBytesObject *b = (PyBytesObject*) o2;
- return Py_SIZE(a) == Py_SIZE(b)
- && *a->ob_sval == *b->ob_sval
- && memcmp(a->ob_sval, b->ob_sval, Py_SIZE(a)) == 0;
-}
-
-static long
-string_hash(PyBytesObject *a)
-{
- register Py_ssize_t len;
- register unsigned char *p;
- register long x;
-
- if (a->ob_shash != -1)
- return a->ob_shash;
- len = Py_SIZE(a);
- p = (unsigned char *) a->ob_sval;
- x = *p << 7;
- while (--len >= 0)
- x = (1000003*x) ^ *p++;
- x ^= Py_SIZE(a);
- if (x == -1)
- x = -2;
- a->ob_shash = x;
- return x;
-}
-
-static PyObject*
-string_subscript(PyBytesObject* self, PyObject* item)
-{
- if (PyIndex_Check(item)) {
- Py_ssize_t i = PyNumber_AsSsize_t(item, PyExc_IndexError);
- if (i == -1 && PyErr_Occurred())
- return NULL;
- if (i < 0)
- i += PyBytes_GET_SIZE(self);
- return string_item(self, i);
- }
- else if (PySlice_Check(item)) {
- Py_ssize_t start, stop, step, slicelength, cur, i;
- char* source_buf;
- char* result_buf;
- PyObject* result;
-
- if (PySlice_GetIndicesEx((PySliceObject*)item,
- PyBytes_GET_SIZE(self),
- &start, &stop, &step, &slicelength) < 0) {
- return NULL;
- }
-
- if (slicelength <= 0) {
- return PyBytes_FromStringAndSize("", 0);
- }
- else if (start == 0 && step == 1 &&
- slicelength == PyBytes_GET_SIZE(self) &&
- PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject *)self;
- }
- else if (step == 1) {
- return PyBytes_FromStringAndSize(
- PyBytes_AS_STRING(self) + start,
- slicelength);
- }
- else {
- source_buf = PyBytes_AsString((PyObject*)self);
- result_buf = (char *)PyMem_Malloc(slicelength);
- if (result_buf == NULL)
- return PyErr_NoMemory();
-
- for (cur = start, i = 0; i < slicelength;
- cur += step, i++) {
- result_buf[i] = source_buf[cur];
- }
-
- result = PyBytes_FromStringAndSize(result_buf,
- slicelength);
- PyMem_Free(result_buf);
- return result;
- }
- }
- else {
- PyErr_Format(PyExc_TypeError,
- "string indices must be integers, not %.200s",
- Py_TYPE(item)->tp_name);
- return NULL;
- }
-}
-
-static Py_ssize_t
-string_buffer_getreadbuf(PyBytesObject *self, Py_ssize_t index, const void **ptr)
-{
- if ( index != 0 ) {
- PyErr_SetString(PyExc_SystemError,
- "accessing non-existent string segment");
- return -1;
- }
- *ptr = (void *)self->ob_sval;
- return Py_SIZE(self);
-}
-
-static Py_ssize_t
-string_buffer_getwritebuf(PyBytesObject *self, Py_ssize_t index, const void **ptr)
-{
- PyErr_SetString(PyExc_TypeError,
- "Cannot use string as modifiable buffer");
- return -1;
-}
-
-static Py_ssize_t
-string_buffer_getsegcount(PyBytesObject *self, Py_ssize_t *lenp)
-{
- if ( lenp )
- *lenp = Py_SIZE(self);
- return 1;
-}
-
-static Py_ssize_t
-string_buffer_getcharbuf(PyBytesObject *self, Py_ssize_t index, const char **ptr)
-{
- if ( index != 0 ) {
- PyErr_SetString(PyExc_SystemError,
- "accessing non-existent string segment");
- return -1;
- }
- *ptr = self->ob_sval;
- return Py_SIZE(self);
-}
-
-static int
-string_buffer_getbuffer(PyBytesObject *self, Py_buffer *view, int flags)
-{
- return PyBuffer_FillInfo(view, (void *)self->ob_sval, Py_SIZE(self),
- 0, flags);
-}
-
-static PySequenceMethods string_as_sequence = {
- (lenfunc)string_length, /*sq_length*/
- (binaryfunc)string_concat, /*sq_concat*/
- (ssizeargfunc)string_repeat, /*sq_repeat*/
- (ssizeargfunc)string_item, /*sq_item*/
- (ssizessizeargfunc)string_slice, /*sq_slice*/
- 0, /*sq_ass_item*/
- 0, /*sq_ass_slice*/
- (objobjproc)string_contains /*sq_contains*/
-};
-
-static PyMappingMethods string_as_mapping = {
- (lenfunc)string_length,
- (binaryfunc)string_subscript,
- 0,
-};
-
-static PyBufferProcs string_as_buffer = {
- (readbufferproc)string_buffer_getreadbuf,
- (writebufferproc)string_buffer_getwritebuf,
- (segcountproc)string_buffer_getsegcount,
- (charbufferproc)string_buffer_getcharbuf,
- (getbufferproc)string_buffer_getbuffer,
- 0, /* XXX */
-};
-
-
-
-#define LEFTSTRIP 0
-#define RIGHTSTRIP 1
-#define BOTHSTRIP 2
-
-/* Arrays indexed by above */
-static const char *stripformat[] = {"|O:lstrip", "|O:rstrip", "|O:strip"};
-
-#define STRIPNAME(i) (stripformat[i]+3)
-
-
-/* Don't call if length < 2 */
-#define Py_STRING_MATCH(target, offset, pattern, length) \
- (target[offset] == pattern[0] && \
- target[offset+length-1] == pattern[length-1] && \
- !memcmp(target+offset+1, pattern+1, length-2) )
-
-
-/* Overallocate the initial list to reduce the number of reallocs for small
- split sizes. Eg, "A A A A A A A A A A".split() (10 elements) has three
- resizes, to sizes 4, 8, then 16. Most observed string splits are for human
- text (roughly 11 words per line) and field delimited data (usually 1-10
- fields). For large strings the split algorithms are bandwidth limited
- so increasing the preallocation likely will not improve things.*/
-
-#define MAX_PREALLOC 12
-
-/* 5 splits gives 6 elements */
-#define PREALLOC_SIZE(maxsplit) \
- (maxsplit >= MAX_PREALLOC ? MAX_PREALLOC : maxsplit+1)
-
-#define SPLIT_APPEND(data, left, right) \
- str = PyBytes_FromStringAndSize((data) + (left), \
- (right) - (left)); \
- if (str == NULL) \
- goto onError; \
- if (PyList_Append(list, str)) { \
- Py_DECREF(str); \
- goto onError; \
- } \
- else \
- Py_DECREF(str);
-
-#define SPLIT_ADD(data, left, right) { \
- str = PyBytes_FromStringAndSize((data) + (left), \
- (right) - (left)); \
- if (str == NULL) \
- goto onError; \
- if (count < MAX_PREALLOC) { \
- PyList_SET_ITEM(list, count, str); \
- } else { \
- if (PyList_Append(list, str)) { \
- Py_DECREF(str); \
- goto onError; \
- } \
- else \
- Py_DECREF(str); \
- } \
- count++; }
-
-/* Always force the list to the expected size. */
-#define FIX_PREALLOC_SIZE(list) Py_SIZE(list) = count
-
-#define SKIP_SPACE(s, i, len) { while (i<len && isspace(Py_CHARMASK(s[i]))) i++; }
-#define SKIP_NONSPACE(s, i, len) { while (i<len && !isspace(Py_CHARMASK(s[i]))) i++; }
-#define RSKIP_SPACE(s, i) { while (i>=0 && isspace(Py_CHARMASK(s[i]))) i--; }
-#define RSKIP_NONSPACE(s, i) { while (i>=0 && !isspace(Py_CHARMASK(s[i]))) i--; }
-
-Py_LOCAL_INLINE(PyObject *)
-split_whitespace(PyBytesObject *self, Py_ssize_t len, Py_ssize_t maxsplit)
-{
- const char *s = PyBytes_AS_STRING(self);
- Py_ssize_t i, j, count=0;
- PyObject *str;
- PyObject *list = PyList_New(PREALLOC_SIZE(maxsplit));
-
- if (list == NULL)
- return NULL;
-
- i = j = 0;
-
- while (maxsplit-- > 0) {
- SKIP_SPACE(s, i, len);
- if (i==len) break;
- j = i; i++;
- SKIP_NONSPACE(s, i, len);
- if (j == 0 && i == len && PyBytes_CheckExact(self)) {
- /* No whitespace in self, so just use it as list[0] */
- Py_INCREF(self);
- PyList_SET_ITEM(list, 0, (PyObject *)self);
- count++;
- break;
- }
- SPLIT_ADD(s, j, i);
- }
-
- if (i < len) {
- /* Only occurs when maxsplit was reached */
- /* Skip any remaining whitespace and copy to end of string */
- SKIP_SPACE(s, i, len);
- if (i != len)
- SPLIT_ADD(s, i, len);
- }
- FIX_PREALLOC_SIZE(list);
- return list;
- onError:
- Py_DECREF(list);
- return NULL;
-}
-
-Py_LOCAL_INLINE(PyObject *)
-split_char(PyBytesObject *self, Py_ssize_t len, char ch, Py_ssize_t maxcount)
-{
- const char *s = PyBytes_AS_STRING(self);
- register Py_ssize_t i, j, count=0;
- PyObject *str;
- PyObject *list = PyList_New(PREALLOC_SIZE(maxcount));
-
- if (list == NULL)
- return NULL;
-
- i = j = 0;
- while ((j < len) && (maxcount-- > 0)) {
- for(; j<len; j++) {
- /* I found that using memchr makes no difference */
- if (s[j] == ch) {
- SPLIT_ADD(s, i, j);
- i = j = j + 1;
- break;
- }
- }
- }
- if (i == 0 && count == 0 && PyBytes_CheckExact(self)) {
- /* ch not in self, so just use self as list[0] */
- Py_INCREF(self);
- PyList_SET_ITEM(list, 0, (PyObject *)self);
- count++;
- }
- else if (i <= len) {
- SPLIT_ADD(s, i, len);
- }
- FIX_PREALLOC_SIZE(list);
- return list;
-
- onError:
- Py_DECREF(list);
- return NULL;
-}
-
-PyDoc_STRVAR(split__doc__,
-"S.split([sep [,maxsplit]]) -> list of strings\n\
-\n\
-Return a list of the words in the string S, using sep as the\n\
-delimiter string. If maxsplit is given, at most maxsplit\n\
-splits are done. If sep is not specified or is None, any\n\
-whitespace string is a separator and empty strings are removed\n\
-from the result.");
-
-static PyObject *
-string_split(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t len = PyBytes_GET_SIZE(self), n, i, j;
- Py_ssize_t maxsplit = -1, count=0;
- const char *s = PyBytes_AS_STRING(self), *sub;
- PyObject *list, *str, *subobj = Py_None;
-#ifdef USE_FAST
- Py_ssize_t pos;
-#endif
-
- if (!PyArg_ParseTuple(args, "|On:split", &subobj, &maxsplit))
- return NULL;
- if (maxsplit < 0)
- maxsplit = PY_SSIZE_T_MAX;
- if (subobj == Py_None)
- return split_whitespace(self, len, maxsplit);
- if (PyBytes_Check(subobj)) {
- sub = PyBytes_AS_STRING(subobj);
- n = PyBytes_GET_SIZE(subobj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(subobj))
- return PyUnicode_Split((PyObject *)self, subobj, maxsplit);
-#endif
- else if (PyObject_AsCharBuffer(subobj, &sub, &n))
- return NULL;
-
- if (n == 0) {
- PyErr_SetString(PyExc_ValueError, "empty separator");
- return NULL;
- }
- else if (n == 1)
- return split_char(self, len, sub[0], maxsplit);
-
- list = PyList_New(PREALLOC_SIZE(maxsplit));
- if (list == NULL)
- return NULL;
-
-#ifdef USE_FAST
- i = j = 0;
- while (maxsplit-- > 0) {
- pos = fastsearch(s+i, len-i, sub, n, FAST_SEARCH);
- if (pos < 0)
- break;
- j = i+pos;
- SPLIT_ADD(s, i, j);
- i = j + n;
- }
-#else
- i = j = 0;
- while ((j+n <= len) && (maxsplit-- > 0)) {
- for (; j+n <= len; j++) {
- if (Py_STRING_MATCH(s, j, sub, n)) {
- SPLIT_ADD(s, i, j);
- i = j = j + n;
- break;
- }
- }
- }
-#endif
- SPLIT_ADD(s, i, len);
- FIX_PREALLOC_SIZE(list);
- return list;
-
- onError:
- Py_DECREF(list);
- return NULL;
-}
-
-PyDoc_STRVAR(partition__doc__,
-"S.partition(sep) -> (head, sep, tail)\n\
-\n\
-Searches for the separator sep in S, and returns the part before it,\n\
-the separator itself, and the part after it. If the separator is not\n\
-found, returns S and two empty strings.");
-
-static PyObject *
-string_partition(PyBytesObject *self, PyObject *sep_obj)
-{
- const char *sep;
- Py_ssize_t sep_len;
-
- if (PyBytes_Check(sep_obj)) {
- sep = PyBytes_AS_STRING(sep_obj);
- sep_len = PyBytes_GET_SIZE(sep_obj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(sep_obj))
- return PyUnicode_Partition((PyObject *) self, sep_obj);
-#endif
- else if (PyObject_AsCharBuffer(sep_obj, &sep, &sep_len))
- return NULL;
-
- return stringlib_partition(
- (PyObject*) self,
- PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
- sep_obj, sep, sep_len
- );
-}
-
-PyDoc_STRVAR(rpartition__doc__,
-"S.rpartition(sep) -> (tail, sep, head)\n\
-\n\
-Searches for the separator sep in S, starting at the end of S, and returns\n\
-the part before it, the separator itself, and the part after it. If the\n\
-separator is not found, returns two empty strings and S.");
-
-static PyObject *
-string_rpartition(PyBytesObject *self, PyObject *sep_obj)
-{
- const char *sep;
- Py_ssize_t sep_len;
-
- if (PyBytes_Check(sep_obj)) {
- sep = PyBytes_AS_STRING(sep_obj);
- sep_len = PyBytes_GET_SIZE(sep_obj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(sep_obj))
- return PyUnicode_Partition((PyObject *) self, sep_obj);
-#endif
- else if (PyObject_AsCharBuffer(sep_obj, &sep, &sep_len))
- return NULL;
-
- return stringlib_rpartition(
- (PyObject*) self,
- PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
- sep_obj, sep, sep_len
- );
-}
-
-Py_LOCAL_INLINE(PyObject *)
-rsplit_whitespace(PyBytesObject *self, Py_ssize_t len, Py_ssize_t maxsplit)
-{
- const char *s = PyBytes_AS_STRING(self);
- Py_ssize_t i, j, count=0;
- PyObject *str;
- PyObject *list = PyList_New(PREALLOC_SIZE(maxsplit));
-
- if (list == NULL)
- return NULL;
-
- i = j = len-1;
-
- while (maxsplit-- > 0) {
- RSKIP_SPACE(s, i);
- if (i<0) break;
- j = i; i--;
- RSKIP_NONSPACE(s, i);
- if (j == len-1 && i < 0 && PyBytes_CheckExact(self)) {
- /* No whitespace in self, so just use it as list[0] */
- Py_INCREF(self);
- PyList_SET_ITEM(list, 0, (PyObject *)self);
- count++;
- break;
- }
- SPLIT_ADD(s, i + 1, j + 1);
- }
- if (i >= 0) {
- /* Only occurs when maxsplit was reached */
- /* Skip any remaining whitespace and copy to beginning of string */
- RSKIP_SPACE(s, i);
- if (i >= 0)
- SPLIT_ADD(s, 0, i + 1);
-
- }
- FIX_PREALLOC_SIZE(list);
- if (PyList_Reverse(list) < 0)
- goto onError;
- return list;
- onError:
- Py_DECREF(list);
- return NULL;
-}
-
-Py_LOCAL_INLINE(PyObject *)
-rsplit_char(PyBytesObject *self, Py_ssize_t len, char ch, Py_ssize_t maxcount)
-{
- const char *s = PyBytes_AS_STRING(self);
- register Py_ssize_t i, j, count=0;
- PyObject *str;
- PyObject *list = PyList_New(PREALLOC_SIZE(maxcount));
-
- if (list == NULL)
- return NULL;
-
- i = j = len - 1;
- while ((i >= 0) && (maxcount-- > 0)) {
- for (; i >= 0; i--) {
- if (s[i] == ch) {
- SPLIT_ADD(s, i + 1, j + 1);
- j = i = i - 1;
- break;
- }
- }
- }
- if (i < 0 && count == 0 && PyBytes_CheckExact(self)) {
- /* ch not in self, so just use self as list[0] */
- Py_INCREF(self);
- PyList_SET_ITEM(list, 0, (PyObject *)self);
- count++;
- }
- else if (j >= -1) {
- SPLIT_ADD(s, 0, j + 1);
- }
- FIX_PREALLOC_SIZE(list);
- if (PyList_Reverse(list) < 0)
- goto onError;
- return list;
-
- onError:
- Py_DECREF(list);
- return NULL;
-}
-
-PyDoc_STRVAR(rsplit__doc__,
-"S.rsplit([sep [,maxsplit]]) -> list of strings\n\
-\n\
-Return a list of the words in the string S, using sep as the\n\
-delimiter string, starting at the end of the string and working\n\
-to the front. If maxsplit is given, at most maxsplit splits are\n\
-done. If sep is not specified or is None, any whitespace string\n\
-is a separator.");
-
-static PyObject *
-string_rsplit(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t len = PyBytes_GET_SIZE(self), n, i, j;
- Py_ssize_t maxsplit = -1, count=0;
- const char *s, *sub;
- PyObject *list, *str, *subobj = Py_None;
-
- if (!PyArg_ParseTuple(args, "|On:rsplit", &subobj, &maxsplit))
- return NULL;
- if (maxsplit < 0)
- maxsplit = PY_SSIZE_T_MAX;
- if (subobj == Py_None)
- return rsplit_whitespace(self, len, maxsplit);
- if (PyBytes_Check(subobj)) {
- sub = PyBytes_AS_STRING(subobj);
- n = PyBytes_GET_SIZE(subobj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(subobj))
- return PyUnicode_RSplit((PyObject *)self, subobj, maxsplit);
-#endif
- else if (PyObject_AsCharBuffer(subobj, &sub, &n))
- return NULL;
-
- if (n == 0) {
- PyErr_SetString(PyExc_ValueError, "empty separator");
- return NULL;
- }
- else if (n == 1)
- return rsplit_char(self, len, sub[0], maxsplit);
-
- list = PyList_New(PREALLOC_SIZE(maxsplit));
- if (list == NULL)
- return NULL;
-
- j = len;
- i = j - n;
-
- s = PyBytes_AS_STRING(self);
- while ( (i >= 0) && (maxsplit-- > 0) ) {
- for (; i>=0; i--) {
- if (Py_STRING_MATCH(s, i, sub, n)) {
- SPLIT_ADD(s, i + n, j);
- j = i;
- i -= n;
- break;
- }
- }
- }
- SPLIT_ADD(s, 0, j);
- FIX_PREALLOC_SIZE(list);
- if (PyList_Reverse(list) < 0)
- goto onError;
- return list;
-
-onError:
- Py_DECREF(list);
- return NULL;
-}
-
-
-PyDoc_STRVAR(join__doc__,
-"S.join(sequence) -> string\n\
-\n\
-Return a string which is the concatenation of the strings in the\n\
-sequence. The separator between elements is S.");
-
-static PyObject *
-string_join(PyBytesObject *self, PyObject *orig)
-{
- char *sep = PyBytes_AS_STRING(self);
- const Py_ssize_t seplen = PyBytes_GET_SIZE(self);
- PyObject *res = NULL;
- char *p;
- Py_ssize_t seqlen = 0;
- size_t sz = 0;
- Py_ssize_t i;
- PyObject *seq, *item;
-
- seq = PySequence_Fast(orig, "");
- if (seq == NULL) {
- return NULL;
- }
-
- seqlen = PySequence_Size(seq);
- if (seqlen == 0) {
- Py_DECREF(seq);
- return PyBytes_FromString("");
- }
- if (seqlen == 1) {
- item = PySequence_Fast_GET_ITEM(seq, 0);
- if (PyBytes_CheckExact(item) || PyUnicode_CheckExact(item)) {
- Py_INCREF(item);
- Py_DECREF(seq);
- return item;
- }
- }
-
- /* There are at least two things to join, or else we have a subclass
- * of the builtin types in the sequence.
- * Do a pre-pass to figure out the total amount of space we'll
- * need (sz), see whether any argument is absurd, and defer to
- * the Unicode join if appropriate.
- */
- for (i = 0; i < seqlen; i++) {
- const size_t old_sz = sz;
- item = PySequence_Fast_GET_ITEM(seq, i);
- if (!PyBytes_Check(item)){
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(item)) {
- /* Defer to Unicode join.
- * CAUTION: There's no gurantee that the
- * original sequence can be iterated over
- * again, so we must pass seq here.
- */
- PyObject *result;
- result = PyUnicode_Join((PyObject *)self, seq);
- Py_DECREF(seq);
- return result;
- }
-#endif
- PyErr_Format(PyExc_TypeError,
- "sequence item %zd: expected string,"
- " %.80s found",
- i, Py_TYPE(item)->tp_name);
- Py_DECREF(seq);
- return NULL;
- }
- sz += PyBytes_GET_SIZE(item);
- if (i != 0)
- sz += seplen;
- if (sz < old_sz || sz > PY_SSIZE_T_MAX) {
- PyErr_SetString(PyExc_OverflowError,
- "join() result is too long for a Python string");
- Py_DECREF(seq);
- return NULL;
- }
- }
-
- /* Allocate result space. */
- res = PyBytes_FromStringAndSize((char*)NULL, sz);
- if (res == NULL) {
- Py_DECREF(seq);
- return NULL;
- }
-
- /* Catenate everything. */
- p = PyBytes_AS_STRING(res);
- for (i = 0; i < seqlen; ++i) {
- size_t n;
- item = PySequence_Fast_GET_ITEM(seq, i);
- n = PyBytes_GET_SIZE(item);
- Py_MEMCPY(p, PyBytes_AS_STRING(item), n);
- p += n;
- if (i < seqlen - 1) {
- Py_MEMCPY(p, sep, seplen);
- p += seplen;
- }
- }
-
- Py_DECREF(seq);
- return res;
-}
-
-PyObject *
-_PyBytes_Join(PyObject *sep, PyObject *x)
-{
- assert(sep != NULL && PyBytes_Check(sep));
- assert(x != NULL);
- return string_join((PyBytesObject *)sep, x);
-}
-
-Py_LOCAL_INLINE(void)
-string_adjust_indices(Py_ssize_t *start, Py_ssize_t *end, Py_ssize_t len)
-{
- if (*end > len)
- *end = len;
- else if (*end < 0)
- *end += len;
- if (*end < 0)
- *end = 0;
- if (*start < 0)
- *start += len;
- if (*start < 0)
- *start = 0;
-}
-
-Py_LOCAL_INLINE(Py_ssize_t)
-string_find_internal(PyBytesObject *self, PyObject *args, int dir)
-{
- PyObject *subobj;
- const char *sub;
- Py_ssize_t sub_len;
- Py_ssize_t start=0, end=PY_SSIZE_T_MAX;
- PyObject *obj_start=Py_None, *obj_end=Py_None;
-
- if (!PyArg_ParseTuple(args, "O|OO:find/rfind/index/rindex", &subobj,
- &obj_start, &obj_end))
- return -2;
- /* To support None in "start" and "end" arguments, meaning
- the same as if they were not passed.
- */
- if (obj_start != Py_None)
- if (!_PyEval_SliceIndex(obj_start, &start))
- return -2;
- if (obj_end != Py_None)
- if (!_PyEval_SliceIndex(obj_end, &end))
- return -2;
-
- if (PyBytes_Check(subobj)) {
- sub = PyBytes_AS_STRING(subobj);
- sub_len = PyBytes_GET_SIZE(subobj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(subobj))
- return PyUnicode_Find(
- (PyObject *)self, subobj, start, end, dir);
-#endif
- else if (PyObject_AsCharBuffer(subobj, &sub, &sub_len))
- /* XXX - the "expected a character buffer object" is pretty
- confusing for a non-expert. remap to something else ? */
- return -2;
-
- if (dir > 0)
- return stringlib_find_slice(
- PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
- sub, sub_len, start, end);
- else
- return stringlib_rfind_slice(
- PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
- sub, sub_len, start, end);
-}
-
-
-PyDoc_STRVAR(find__doc__,
-"S.find(sub [,start [,end]]) -> int\n\
-\n\
-Return the lowest index in S where substring sub is found,\n\
-such that sub is contained within s[start:end]. Optional\n\
-arguments start and end are interpreted as in slice notation.\n\
-\n\
-Return -1 on failure.");
-
-static PyObject *
-string_find(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t result = string_find_internal(self, args, +1);
- if (result == -2)
- return NULL;
- return PyInt_FromSsize_t(result);
-}
-
-
-PyDoc_STRVAR(index__doc__,
-"S.index(sub [,start [,end]]) -> int\n\
-\n\
-Like S.find() but raise ValueError when the substring is not found.");
-
-static PyObject *
-string_index(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t result = string_find_internal(self, args, +1);
- if (result == -2)
- return NULL;
- if (result == -1) {
- PyErr_SetString(PyExc_ValueError,
- "substring not found");
- return NULL;
- }
- return PyInt_FromSsize_t(result);
-}
-
-
-PyDoc_STRVAR(rfind__doc__,
-"S.rfind(sub [,start [,end]]) -> int\n\
-\n\
-Return the highest index in S where substring sub is found,\n\
-such that sub is contained within s[start:end]. Optional\n\
-arguments start and end are interpreted as in slice notation.\n\
-\n\
-Return -1 on failure.");
-
-static PyObject *
-string_rfind(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t result = string_find_internal(self, args, -1);
- if (result == -2)
- return NULL;
- return PyInt_FromSsize_t(result);
-}
-
-
-PyDoc_STRVAR(rindex__doc__,
-"S.rindex(sub [,start [,end]]) -> int\n\
-\n\
-Like S.rfind() but raise ValueError when the substring is not found.");
-
-static PyObject *
-string_rindex(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t result = string_find_internal(self, args, -1);
- if (result == -2)
- return NULL;
- if (result == -1) {
- PyErr_SetString(PyExc_ValueError,
- "substring not found");
- return NULL;
- }
- return PyInt_FromSsize_t(result);
-}
-
-
-Py_LOCAL_INLINE(PyObject *)
-do_xstrip(PyBytesObject *self, int striptype, PyObject *sepobj)
-{
- char *s = PyBytes_AS_STRING(self);
- Py_ssize_t len = PyBytes_GET_SIZE(self);
- char *sep = PyBytes_AS_STRING(sepobj);
- Py_ssize_t seplen = PyBytes_GET_SIZE(sepobj);
- Py_ssize_t i, j;
-
- i = 0;
- if (striptype != RIGHTSTRIP) {
- while (i < len && memchr(sep, Py_CHARMASK(s[i]), seplen)) {
- i++;
- }
- }
-
- j = len;
- if (striptype != LEFTSTRIP) {
- do {
- j--;
- } while (j >= i && memchr(sep, Py_CHARMASK(s[j]), seplen));
- j++;
- }
-
- if (i == 0 && j == len && PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject*)self;
- }
- else
- return PyBytes_FromStringAndSize(s+i, j-i);
-}
-
-
-Py_LOCAL_INLINE(PyObject *)
-do_strip(PyBytesObject *self, int striptype)
-{
- char *s = PyBytes_AS_STRING(self);
- Py_ssize_t len = PyBytes_GET_SIZE(self), i, j;
-
- i = 0;
- if (striptype != RIGHTSTRIP) {
- while (i < len && isspace(Py_CHARMASK(s[i]))) {
- i++;
- }
- }
-
- j = len;
- if (striptype != LEFTSTRIP) {
- do {
- j--;
- } while (j >= i && isspace(Py_CHARMASK(s[j])));
- j++;
- }
-
- if (i == 0 && j == len && PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject*)self;
- }
- else
- return PyBytes_FromStringAndSize(s+i, j-i);
-}
-
-
-Py_LOCAL_INLINE(PyObject *)
-do_argstrip(PyBytesObject *self, int striptype, PyObject *args)
-{
- PyObject *sep = NULL;
-
- if (!PyArg_ParseTuple(args, (char *)stripformat[striptype], &sep))
- return NULL;
-
- if (sep != NULL && sep != Py_None) {
- if (PyBytes_Check(sep))
- return do_xstrip(self, striptype, sep);
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(sep)) {
- PyObject *uniself = PyUnicode_FromObject((PyObject *)self);
- PyObject *res;
- if (uniself==NULL)
- return NULL;
- res = _PyUnicode_XStrip((PyUnicodeObject *)uniself,
- striptype, sep);
- Py_DECREF(uniself);
- return res;
- }
-#endif
- PyErr_Format(PyExc_TypeError,
-#ifdef Py_USING_UNICODE
- "%s arg must be None, str or unicode",
-#else
- "%s arg must be None or str",
-#endif
- STRIPNAME(striptype));
- return NULL;
- }
-
- return do_strip(self, striptype);
-}
-
-
-PyDoc_STRVAR(strip__doc__,
-"S.strip([chars]) -> string or unicode\n\
-\n\
-Return a copy of the string S with leading and trailing\n\
-whitespace removed.\n\
-If chars is given and not None, remove characters in chars instead.\n\
-If chars is unicode, S will be converted to unicode before stripping");
-
-static PyObject *
-string_strip(PyBytesObject *self, PyObject *args)
-{
- if (PyTuple_GET_SIZE(args) == 0)
- return do_strip(self, BOTHSTRIP); /* Common case */
- else
- return do_argstrip(self, BOTHSTRIP, args);
-}
-
-
-PyDoc_STRVAR(lstrip__doc__,
-"S.lstrip([chars]) -> string or unicode\n\
-\n\
-Return a copy of the string S with leading whitespace removed.\n\
-If chars is given and not None, remove characters in chars instead.\n\
-If chars is unicode, S will be converted to unicode before stripping");
-
-static PyObject *
-string_lstrip(PyBytesObject *self, PyObject *args)
-{
- if (PyTuple_GET_SIZE(args) == 0)
- return do_strip(self, LEFTSTRIP); /* Common case */
- else
- return do_argstrip(self, LEFTSTRIP, args);
-}
-
-
-PyDoc_STRVAR(rstrip__doc__,
-"S.rstrip([chars]) -> string or unicode\n\
-\n\
-Return a copy of the string S with trailing whitespace removed.\n\
-If chars is given and not None, remove characters in chars instead.\n\
-If chars is unicode, S will be converted to unicode before stripping");
-
-static PyObject *
-string_rstrip(PyBytesObject *self, PyObject *args)
-{
- if (PyTuple_GET_SIZE(args) == 0)
- return do_strip(self, RIGHTSTRIP); /* Common case */
- else
- return do_argstrip(self, RIGHTSTRIP, args);
-}
-
-
-PyDoc_STRVAR(lower__doc__,
-"S.lower() -> string\n\
-\n\
-Return a copy of the string S converted to lowercase.");
-
-/* _tolower and _toupper are defined by SUSv2, but they're not ISO C */
-#ifndef _tolower
-#define _tolower tolower
-#endif
-
-static PyObject *
-string_lower(PyBytesObject *self)
-{
- char *s;
- Py_ssize_t i, n = PyBytes_GET_SIZE(self);
- PyObject *newobj;
-
- newobj = PyBytes_FromStringAndSize(NULL, n);
- if (!newobj)
- return NULL;
-
- s = PyBytes_AS_STRING(newobj);
-
- Py_MEMCPY(s, PyBytes_AS_STRING(self), n);
-
- for (i = 0; i < n; i++) {
- int c = Py_CHARMASK(s[i]);
- if (isupper(c))
- s[i] = _tolower(c);
- }
-
- return newobj;
-}
-
-PyDoc_STRVAR(upper__doc__,
-"S.upper() -> string\n\
-\n\
-Return a copy of the string S converted to uppercase.");
-
-#ifndef _toupper
-#define _toupper toupper
-#endif
-
-static PyObject *
-string_upper(PyBytesObject *self)
-{
- char *s;
- Py_ssize_t i, n = PyBytes_GET_SIZE(self);
- PyObject *newobj;
-
- newobj = PyBytes_FromStringAndSize(NULL, n);
- if (!newobj)
- return NULL;
-
- s = PyBytes_AS_STRING(newobj);
-
- Py_MEMCPY(s, PyBytes_AS_STRING(self), n);
-
- for (i = 0; i < n; i++) {
- int c = Py_CHARMASK(s[i]);
- if (islower(c))
- s[i] = _toupper(c);
- }
-
- return newobj;
-}
-
-PyDoc_STRVAR(title__doc__,
-"S.title() -> string\n\
-\n\
-Return a titlecased version of S, i.e. words start with uppercase\n\
-characters, all remaining cased characters have lowercase.");
-
-static PyObject*
-string_title(PyBytesObject *self)
-{
- char *s = PyBytes_AS_STRING(self), *s_new;
- Py_ssize_t i, n = PyBytes_GET_SIZE(self);
- int previous_is_cased = 0;
- PyObject *newobj;
-
- newobj = PyBytes_FromStringAndSize(NULL, n);
- if (newobj == NULL)
- return NULL;
- s_new = PyBytes_AsString(newobj);
- for (i = 0; i < n; i++) {
- int c = Py_CHARMASK(*s++);
- if (islower(c)) {
- if (!previous_is_cased)
- c = toupper(c);
- previous_is_cased = 1;
- } else if (isupper(c)) {
- if (previous_is_cased)
- c = tolower(c);
- previous_is_cased = 1;
- } else
- previous_is_cased = 0;
- *s_new++ = c;
- }
- return newobj;
-}
-
-PyDoc_STRVAR(capitalize__doc__,
-"S.capitalize() -> string\n\
-\n\
-Return a copy of the string S with only its first character\n\
-capitalized.");
-
-static PyObject *
-string_capitalize(PyBytesObject *self)
-{
- char *s = PyBytes_AS_STRING(self), *s_new;
- Py_ssize_t i, n = PyBytes_GET_SIZE(self);
- PyObject *newobj;
-
- newobj = PyBytes_FromStringAndSize(NULL, n);
- if (newobj == NULL)
- return NULL;
- s_new = PyBytes_AsString(newobj);
- if (0 < n) {
- int c = Py_CHARMASK(*s++);
- if (islower(c))
- *s_new = toupper(c);
- else
- *s_new = c;
- s_new++;
- }
- for (i = 1; i < n; i++) {
- int c = Py_CHARMASK(*s++);
- if (isupper(c))
- *s_new = tolower(c);
- else
- *s_new = c;
- s_new++;
- }
- return newobj;
-}
-
-
-PyDoc_STRVAR(count__doc__,
-"S.count(sub[, start[, end]]) -> int\n\
-\n\
-Return the number of non-overlapping occurrences of substring sub in\n\
-string S[start:end]. Optional arguments start and end are interpreted\n\
-as in slice notation.");
-
-static PyObject *
-string_count(PyBytesObject *self, PyObject *args)
-{
- PyObject *sub_obj;
- const char *str = PyBytes_AS_STRING(self), *sub;
- Py_ssize_t sub_len;
- Py_ssize_t start = 0, end = PY_SSIZE_T_MAX;
-
- if (!PyArg_ParseTuple(args, "O|O&O&:count", &sub_obj,
- _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end))
- return NULL;
-
- if (PyBytes_Check(sub_obj)) {
- sub = PyBytes_AS_STRING(sub_obj);
- sub_len = PyBytes_GET_SIZE(sub_obj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(sub_obj)) {
- Py_ssize_t count;
- count = PyUnicode_Count((PyObject *)self, sub_obj, start, end);
- if (count == -1)
- return NULL;
- else
- return PyInt_FromSsize_t(count);
- }
-#endif
- else if (PyObject_AsCharBuffer(sub_obj, &sub, &sub_len))
- return NULL;
-
- string_adjust_indices(&start, &end, PyBytes_GET_SIZE(self));
-
- return PyInt_FromSsize_t(
- stringlib_count(str + start, end - start, sub, sub_len)
- );
-}
-
-PyDoc_STRVAR(swapcase__doc__,
-"S.swapcase() -> string\n\
-\n\
-Return a copy of the string S with uppercase characters\n\
-converted to lowercase and vice versa.");
-
-static PyObject *
-string_swapcase(PyBytesObject *self)
-{
- char *s = PyBytes_AS_STRING(self), *s_new;
- Py_ssize_t i, n = PyBytes_GET_SIZE(self);
- PyObject *newobj;
-
- newobj = PyBytes_FromStringAndSize(NULL, n);
- if (newobj == NULL)
- return NULL;
- s_new = PyBytes_AsString(newobj);
- for (i = 0; i < n; i++) {
- int c = Py_CHARMASK(*s++);
- if (islower(c)) {
- *s_new = toupper(c);
- }
- else if (isupper(c)) {
- *s_new = tolower(c);
- }
- else
- *s_new = c;
- s_new++;
- }
- return newobj;
-}
-
-
-PyDoc_STRVAR(translate__doc__,
-"S.translate(table [,deletechars]) -> string\n\
-\n\
-Return a copy of the string S, where all characters occurring\n\
-in the optional argument deletechars are removed, and the\n\
-remaining characters have been mapped through the given\n\
-translation table, which must be a string of length 256.");
-
-static PyObject *
-string_translate(PyBytesObject *self, PyObject *args)
-{
- register char *input, *output;
- const char *table;
- register Py_ssize_t i, c, changed = 0;
- PyObject *input_obj = (PyObject*)self;
- const char *output_start, *del_table=NULL;
- Py_ssize_t inlen, tablen, dellen = 0;
- PyObject *result;
- int trans_table[256];
- PyObject *tableobj, *delobj = NULL;
-
- if (!PyArg_UnpackTuple(args, "translate", 1, 2,
- &tableobj, &delobj))
- return NULL;
-
- if (PyBytes_Check(tableobj)) {
- table = PyBytes_AS_STRING(tableobj);
- tablen = PyBytes_GET_SIZE(tableobj);
- }
- else if (tableobj == Py_None) {
- table = NULL;
- tablen = 256;
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(tableobj)) {
- /* Unicode .translate() does not support the deletechars
- parameter; instead a mapping to None will cause characters
- to be deleted. */
- if (delobj != NULL) {
- PyErr_SetString(PyExc_TypeError,
- "deletions are implemented differently for unicode");
- return NULL;
- }
- return PyUnicode_Translate((PyObject *)self, tableobj, NULL);
- }
-#endif
- else if (PyObject_AsCharBuffer(tableobj, &table, &tablen))
- return NULL;
-
- if (tablen != 256) {
- PyErr_SetString(PyExc_ValueError,
- "translation table must be 256 characters long");
- return NULL;
- }
-
- if (delobj != NULL) {
- if (PyBytes_Check(delobj)) {
- del_table = PyBytes_AS_STRING(delobj);
- dellen = PyBytes_GET_SIZE(delobj);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(delobj)) {
- PyErr_SetString(PyExc_TypeError,
- "deletions are implemented differently for unicode");
- return NULL;
- }
-#endif
- else if (PyObject_AsCharBuffer(delobj, &del_table, &dellen))
- return NULL;
- }
- else {
- del_table = NULL;
- dellen = 0;
- }
-
- inlen = PyBytes_GET_SIZE(input_obj);
- result = PyBytes_FromStringAndSize((char *)NULL, inlen);
- if (result == NULL)
- return NULL;
- output_start = output = PyBytes_AsString(result);
- input = PyBytes_AS_STRING(input_obj);
-
- if (dellen == 0 && table != NULL) {
- /* If no deletions are required, use faster code */
- for (i = inlen; --i >= 0; ) {
- c = Py_CHARMASK(*input++);
- if (Py_CHARMASK((*output++ = table[c])) != c)
- changed = 1;
- }
- if (changed || !PyBytes_CheckExact(input_obj))
- return result;
- Py_DECREF(result);
- Py_INCREF(input_obj);
- return input_obj;
- }
-
- if (table == NULL) {
- for (i = 0; i < 256; i++)
- trans_table[i] = Py_CHARMASK(i);
- } else {
- for (i = 0; i < 256; i++)
- trans_table[i] = Py_CHARMASK(table[i]);
- }
-
- for (i = 0; i < dellen; i++)
- trans_table[(int) Py_CHARMASK(del_table[i])] = -1;
-
- for (i = inlen; --i >= 0; ) {
- c = Py_CHARMASK(*input++);
- if (trans_table[c] != -1)
- if (Py_CHARMASK(*output++ = (char)trans_table[c]) == c)
- continue;
- changed = 1;
- }
- if (!changed && PyBytes_CheckExact(input_obj)) {
- Py_DECREF(result);
- Py_INCREF(input_obj);
- return input_obj;
- }
- /* Fix the size of the resulting string */
- if (inlen > 0)
- _PyBytes_Resize(&result, output - output_start);
- return result;
-}
-
-
-#define FORWARD 1
-#define REVERSE -1
-
-/* find and count characters and substrings */
-
-#define findchar(target, target_len, c) \
- ((char *)memchr((const void *)(target), c, target_len))
-
-/* String ops must return a string. */
-/* If the object is subclass of string, create a copy */
-Py_LOCAL(PyBytesObject *)
-return_self(PyBytesObject *self)
-{
- if (PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return self;
- }
- return (PyBytesObject *)PyBytes_FromStringAndSize(
- PyBytes_AS_STRING(self),
- PyBytes_GET_SIZE(self));
-}
-
-Py_LOCAL_INLINE(Py_ssize_t)
-countchar(const char *target, int target_len, char c, Py_ssize_t maxcount)
-{
- Py_ssize_t count=0;
- const char *start=target;
- const char *end=target+target_len;
-
- while ( (start=findchar(start, end-start, c)) != NULL ) {
- count++;
- if (count >= maxcount)
- break;
- start += 1;
- }
- return count;
-}
-
-Py_LOCAL(Py_ssize_t)
-findstring(const char *target, Py_ssize_t target_len,
- const char *pattern, Py_ssize_t pattern_len,
- Py_ssize_t start,
- Py_ssize_t end,
- int direction)
-{
- if (start < 0) {
- start += target_len;
- if (start < 0)
- start = 0;
- }
- if (end > target_len) {
- end = target_len;
- } else if (end < 0) {
- end += target_len;
- if (end < 0)
- end = 0;
- }
-
- /* zero-length substrings always match at the first attempt */
- if (pattern_len == 0)
- return (direction > 0) ? start : end;
-
- end -= pattern_len;
-
- if (direction < 0) {
- for (; end >= start; end--)
- if (Py_STRING_MATCH(target, end, pattern, pattern_len))
- return end;
- } else {
- for (; start <= end; start++)
- if (Py_STRING_MATCH(target, start, pattern, pattern_len))
- return start;
- }
- return -1;
-}
-
-Py_LOCAL_INLINE(Py_ssize_t)
-countstring(const char *target, Py_ssize_t target_len,
- const char *pattern, Py_ssize_t pattern_len,
- Py_ssize_t start,
- Py_ssize_t end,
- int direction, Py_ssize_t maxcount)
-{
- Py_ssize_t count=0;
-
- if (start < 0) {
- start += target_len;
- if (start < 0)
- start = 0;
- }
- if (end > target_len) {
- end = target_len;
- } else if (end < 0) {
- end += target_len;
- if (end < 0)
- end = 0;
- }
-
- /* zero-length substrings match everywhere */
- if (pattern_len == 0 || maxcount == 0) {
- if (target_len+1 < maxcount)
- return target_len+1;
- return maxcount;
- }
-
- end -= pattern_len;
- if (direction < 0) {
- for (; (end >= start); end--)
- if (Py_STRING_MATCH(target, end, pattern, pattern_len)) {
- count++;
- if (--maxcount <= 0) break;
- end -= pattern_len-1;
- }
- } else {
- for (; (start <= end); start++)
- if (Py_STRING_MATCH(target, start, pattern, pattern_len)) {
- count++;
- if (--maxcount <= 0)
- break;
- start += pattern_len-1;
- }
- }
- return count;
-}
-
-
-/* Algorithms for different cases of string replacement */
-
-/* len(self)>=1, from="", len(to)>=1, maxcount>=1 */
-Py_LOCAL(PyBytesObject *)
-replace_interleave(PyBytesObject *self,
- const char *to_s, Py_ssize_t to_len,
- Py_ssize_t maxcount)
-{
- char *self_s, *result_s;
- Py_ssize_t self_len, result_len;
- Py_ssize_t count, i, product;
- PyBytesObject *result;
-
- self_len = PyBytes_GET_SIZE(self);
-
- /* 1 at the end plus 1 after every character */
- count = self_len+1;
- if (maxcount < count)
- count = maxcount;
-
- /* Check for overflow */
- /* result_len = count * to_len + self_len; */
- product = count * to_len;
- if (product / to_len != count) {
- PyErr_SetString(PyExc_OverflowError,
- "replace string is too long");
- return NULL;
- }
- result_len = product + self_len;
- if (result_len < 0) {
- PyErr_SetString(PyExc_OverflowError,
- "replace string is too long");
- return NULL;
- }
-
- if (! (result = (PyBytesObject *)
- PyBytes_FromStringAndSize(NULL, result_len)) )
- return NULL;
-
- self_s = PyBytes_AS_STRING(self);
- result_s = PyBytes_AS_STRING(result);
-
- /* TODO: special case single character, which doesn't need memcpy */
-
- /* Lay the first one down (guaranteed this will occur) */
- Py_MEMCPY(result_s, to_s, to_len);
- result_s += to_len;
- count -= 1;
-
- for (i=0; i<count; i++) {
- *result_s++ = *self_s++;
- Py_MEMCPY(result_s, to_s, to_len);
- result_s += to_len;
- }
-
- /* Copy the rest of the original string */
- Py_MEMCPY(result_s, self_s, self_len-i);
-
- return result;
-}
-
-/* Special case for deleting a single character */
-/* len(self)>=1, len(from)==1, to="", maxcount>=1 */
-Py_LOCAL(PyBytesObject *)
-replace_delete_single_character(PyBytesObject *self,
- char from_c, Py_ssize_t maxcount)
-{
- char *self_s, *result_s;
- char *start, *next, *end;
- Py_ssize_t self_len, result_len;
- Py_ssize_t count;
- PyBytesObject *result;
-
- self_len = PyBytes_GET_SIZE(self);
- self_s = PyBytes_AS_STRING(self);
-
- count = countchar(self_s, self_len, from_c, maxcount);
- if (count == 0) {
- return return_self(self);
- }
-
- result_len = self_len - count; /* from_len == 1 */
- assert(result_len>=0);
-
- if ( (result = (PyBytesObject *)
- PyBytes_FromStringAndSize(NULL, result_len)) == NULL)
- return NULL;
- result_s = PyBytes_AS_STRING(result);
-
- start = self_s;
- end = self_s + self_len;
- while (count-- > 0) {
- next = findchar(start, end-start, from_c);
- if (next == NULL)
- break;
- Py_MEMCPY(result_s, start, next-start);
- result_s += (next-start);
- start = next+1;
- }
- Py_MEMCPY(result_s, start, end-start);
-
- return result;
-}
-
-/* len(self)>=1, len(from)>=2, to="", maxcount>=1 */
-
-Py_LOCAL(PyBytesObject *)
-replace_delete_substring(PyBytesObject *self,
- const char *from_s, Py_ssize_t from_len,
- Py_ssize_t maxcount) {
- char *self_s, *result_s;
- char *start, *next, *end;
- Py_ssize_t self_len, result_len;
- Py_ssize_t count, offset;
- PyBytesObject *result;
-
- self_len = PyBytes_GET_SIZE(self);
- self_s = PyBytes_AS_STRING(self);
-
- count = countstring(self_s, self_len,
- from_s, from_len,
- 0, self_len, 1,
- maxcount);
-
- if (count == 0) {
- /* no matches */
- return return_self(self);
- }
-
- result_len = self_len - (count * from_len);
- assert (result_len>=0);
-
- if ( (result = (PyBytesObject *)
- PyBytes_FromStringAndSize(NULL, result_len)) == NULL )
- return NULL;
-
- result_s = PyBytes_AS_STRING(result);
-
- start = self_s;
- end = self_s + self_len;
- while (count-- > 0) {
- offset = findstring(start, end-start,
- from_s, from_len,
- 0, end-start, FORWARD);
- if (offset == -1)
- break;
- next = start + offset;
-
- Py_MEMCPY(result_s, start, next-start);
-
- result_s += (next-start);
- start = next+from_len;
- }
- Py_MEMCPY(result_s, start, end-start);
- return result;
-}
-
-/* len(self)>=1, len(from)==len(to)==1, maxcount>=1 */
-Py_LOCAL(PyBytesObject *)
-replace_single_character_in_place(PyBytesObject *self,
- char from_c, char to_c,
- Py_ssize_t maxcount)
-{
- char *self_s, *result_s, *start, *end, *next;
- Py_ssize_t self_len;
- PyBytesObject *result;
-
- /* The result string will be the same size */
- self_s = PyBytes_AS_STRING(self);
- self_len = PyBytes_GET_SIZE(self);
-
- next = findchar(self_s, self_len, from_c);
-
- if (next == NULL) {
- /* No matches; return the original string */
- return return_self(self);
- }
-
- /* Need to make a new string */
- result = (PyBytesObject *) PyBytes_FromStringAndSize(NULL, self_len);
- if (result == NULL)
- return NULL;
- result_s = PyBytes_AS_STRING(result);
- Py_MEMCPY(result_s, self_s, self_len);
-
- /* change everything in-place, starting with this one */
- start = result_s + (next-self_s);
- *start = to_c;
- start++;
- end = result_s + self_len;
-
- while (--maxcount > 0) {
- next = findchar(start, end-start, from_c);
- if (next == NULL)
- break;
- *next = to_c;
- start = next+1;
- }
-
- return result;
-}
-
-/* len(self)>=1, len(from)==len(to)>=2, maxcount>=1 */
-Py_LOCAL(PyBytesObject *)
-replace_substring_in_place(PyBytesObject *self,
- const char *from_s, Py_ssize_t from_len,
- const char *to_s, Py_ssize_t to_len,
- Py_ssize_t maxcount)
-{
- char *result_s, *start, *end;
- char *self_s;
- Py_ssize_t self_len, offset;
- PyBytesObject *result;
-
- /* The result string will be the same size */
-
- self_s = PyBytes_AS_STRING(self);
- self_len = PyBytes_GET_SIZE(self);
-
- offset = findstring(self_s, self_len,
- from_s, from_len,
- 0, self_len, FORWARD);
- if (offset == -1) {
- /* No matches; return the original string */
- return return_self(self);
- }
-
- /* Need to make a new string */
- result = (PyBytesObject *) PyBytes_FromStringAndSize(NULL, self_len);
- if (result == NULL)
- return NULL;
- result_s = PyBytes_AS_STRING(result);
- Py_MEMCPY(result_s, self_s, self_len);
-
- /* change everything in-place, starting with this one */
- start = result_s + offset;
- Py_MEMCPY(start, to_s, from_len);
- start += from_len;
- end = result_s + self_len;
-
- while ( --maxcount > 0) {
- offset = findstring(start, end-start,
- from_s, from_len,
- 0, end-start, FORWARD);
- if (offset==-1)
- break;
- Py_MEMCPY(start+offset, to_s, from_len);
- start += offset+from_len;
- }
-
- return result;
-}
-
-/* len(self)>=1, len(from)==1, len(to)>=2, maxcount>=1 */
-Py_LOCAL(PyBytesObject *)
-replace_single_character(PyBytesObject *self,
- char from_c,
- const char *to_s, Py_ssize_t to_len,
- Py_ssize_t maxcount)
-{
- char *self_s, *result_s;
- char *start, *next, *end;
- Py_ssize_t self_len, result_len;
- Py_ssize_t count, product;
- PyBytesObject *result;
-
- self_s = PyBytes_AS_STRING(self);
- self_len = PyBytes_GET_SIZE(self);
-
- count = countchar(self_s, self_len, from_c, maxcount);
- if (count == 0) {
- /* no matches, return unchanged */
- return return_self(self);
- }
-
- /* use the difference between current and new, hence the "-1" */
- /* result_len = self_len + count * (to_len-1) */
- product = count * (to_len-1);
- if (product / (to_len-1) != count) {
- PyErr_SetString(PyExc_OverflowError, "replace string is too long");
- return NULL;
- }
- result_len = self_len + product;
- if (result_len < 0) {
- PyErr_SetString(PyExc_OverflowError, "replace string is too long");
- return NULL;
- }
-
- if ( (result = (PyBytesObject *)
- PyBytes_FromStringAndSize(NULL, result_len)) == NULL)
- return NULL;
- result_s = PyBytes_AS_STRING(result);
-
- start = self_s;
- end = self_s + self_len;
- while (count-- > 0) {
- next = findchar(start, end-start, from_c);
- if (next == NULL)
- break;
-
- if (next == start) {
- /* replace with the 'to' */
- Py_MEMCPY(result_s, to_s, to_len);
- result_s += to_len;
- start += 1;
- } else {
- /* copy the unchanged old then the 'to' */
- Py_MEMCPY(result_s, start, next-start);
- result_s += (next-start);
- Py_MEMCPY(result_s, to_s, to_len);
- result_s += to_len;
- start = next+1;
- }
- }
- /* Copy the remainder of the remaining string */
- Py_MEMCPY(result_s, start, end-start);
-
- return result;
-}
-
-/* len(self)>=1, len(from)>=2, len(to)>=2, maxcount>=1 */
-Py_LOCAL(PyBytesObject *)
-replace_substring(PyBytesObject *self,
- const char *from_s, Py_ssize_t from_len,
- const char *to_s, Py_ssize_t to_len,
- Py_ssize_t maxcount) {
- char *self_s, *result_s;
- char *start, *next, *end;
- Py_ssize_t self_len, result_len;
- Py_ssize_t count, offset, product;
- PyBytesObject *result;
-
- self_s = PyBytes_AS_STRING(self);
- self_len = PyBytes_GET_SIZE(self);
-
- count = countstring(self_s, self_len,
- from_s, from_len,
- 0, self_len, FORWARD, maxcount);
- if (count == 0) {
- /* no matches, return unchanged */
- return return_self(self);
- }
-
- /* Check for overflow */
- /* result_len = self_len + count * (to_len-from_len) */
- product = count * (to_len-from_len);
- if (product / (to_len-from_len) != count) {
- PyErr_SetString(PyExc_OverflowError, "replace string is too long");
- return NULL;
- }
- result_len = self_len + product;
- if (result_len < 0) {
- PyErr_SetString(PyExc_OverflowError, "replace string is too long");
- return NULL;
- }
-
- if ( (result = (PyBytesObject *)
- PyBytes_FromStringAndSize(NULL, result_len)) == NULL)
- return NULL;
- result_s = PyBytes_AS_STRING(result);
-
- start = self_s;
- end = self_s + self_len;
- while (count-- > 0) {
- offset = findstring(start, end-start,
- from_s, from_len,
- 0, end-start, FORWARD);
- if (offset == -1)
- break;
- next = start+offset;
- if (next == start) {
- /* replace with the 'to' */
- Py_MEMCPY(result_s, to_s, to_len);
- result_s += to_len;
- start += from_len;
- } else {
- /* copy the unchanged old then the 'to' */
- Py_MEMCPY(result_s, start, next-start);
- result_s += (next-start);
- Py_MEMCPY(result_s, to_s, to_len);
- result_s += to_len;
- start = next+from_len;
- }
- }
- /* Copy the remainder of the remaining string */
- Py_MEMCPY(result_s, start, end-start);
-
- return result;
-}
-
-
-Py_LOCAL(PyBytesObject *)
-replace(PyBytesObject *self,
- const char *from_s, Py_ssize_t from_len,
- const char *to_s, Py_ssize_t to_len,
- Py_ssize_t maxcount)
-{
- if (maxcount < 0) {
- maxcount = PY_SSIZE_T_MAX;
- } else if (maxcount == 0 || PyBytes_GET_SIZE(self) == 0) {
- /* nothing to do; return the original string */
- return return_self(self);
- }
-
- if (maxcount == 0 ||
- (from_len == 0 && to_len == 0)) {
- /* nothing to do; return the original string */
- return return_self(self);
- }
-
- /* Handle zero-length special cases */
-
- if (from_len == 0) {
- /* insert the 'to' string everywhere. */
- /* >>> "Python".replace("", ".") */
- /* '.P.y.t.h.o.n.' */
- return replace_interleave(self, to_s, to_len, maxcount);
- }
-
- /* Except for "".replace("", "A") == "A" there is no way beyond this */
- /* point for an empty self string to generate a non-empty string */
- /* Special case so the remaining code always gets a non-empty string */
- if (PyBytes_GET_SIZE(self) == 0) {
- return return_self(self);
- }
-
- if (to_len == 0) {
- /* delete all occurances of 'from' string */
- if (from_len == 1) {
- return replace_delete_single_character(
- self, from_s[0], maxcount);
- } else {
- return replace_delete_substring(self, from_s, from_len, maxcount);
- }
- }
-
- /* Handle special case where both strings have the same length */
-
- if (from_len == to_len) {
- if (from_len == 1) {
- return replace_single_character_in_place(
- self,
- from_s[0],
- to_s[0],
- maxcount);
- } else {
- return replace_substring_in_place(
- self, from_s, from_len, to_s, to_len, maxcount);
- }
- }
-
- /* Otherwise use the more generic algorithms */
- if (from_len == 1) {
- return replace_single_character(self, from_s[0],
- to_s, to_len, maxcount);
- } else {
- /* len('from')>=2, len('to')>=1 */
- return replace_substring(self, from_s, from_len, to_s, to_len, maxcount);
- }
-}
-
-PyDoc_STRVAR(replace__doc__,
-"S.replace (old, new[, count]) -> string\n\
-\n\
-Return a copy of string S with all occurrences of substring\n\
-old replaced by new. If the optional argument count is\n\
-given, only the first count occurrences are replaced.");
-
-static PyObject *
-string_replace(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t count = -1;
- PyObject *from, *to;
- const char *from_s, *to_s;
- Py_ssize_t from_len, to_len;
-
- if (!PyArg_ParseTuple(args, "OO|n:replace", &from, &to, &count))
- return NULL;
-
- if (PyBytes_Check(from)) {
- from_s = PyBytes_AS_STRING(from);
- from_len = PyBytes_GET_SIZE(from);
- }
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(from))
- return PyUnicode_Replace((PyObject *)self,
- from, to, count);
-#endif
- else if (PyObject_AsCharBuffer(from, &from_s, &from_len))
- return NULL;
-
- if (PyBytes_Check(to)) {
- to_s = PyBytes_AS_STRING(to);
- to_len = PyBytes_GET_SIZE(to);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(to))
- return PyUnicode_Replace((PyObject *)self,
- from, to, count);
-#endif
- else if (PyObject_AsCharBuffer(to, &to_s, &to_len))
- return NULL;
-
- return (PyObject *)replace((PyBytesObject *) self,
- from_s, from_len,
- to_s, to_len, count);
-}
-
-/** End DALKE **/
-
-/* Matches the end (direction >= 0) or start (direction < 0) of self
- * against substr, using the start and end arguments. Returns
- * -1 on error, 0 if not found and 1 if found.
- */
-Py_LOCAL(int)
-_string_tailmatch(PyBytesObject *self, PyObject *substr, Py_ssize_t start,
- Py_ssize_t end, int direction)
-{
- Py_ssize_t len = PyBytes_GET_SIZE(self);
- Py_ssize_t slen;
- const char* sub;
- const char* str;
-
- if (PyBytes_Check(substr)) {
- sub = PyBytes_AS_STRING(substr);
- slen = PyBytes_GET_SIZE(substr);
- }
-#ifdef Py_USING_UNICODE
- else if (PyUnicode_Check(substr))
- return PyUnicode_Tailmatch((PyObject *)self,
- substr, start, end, direction);
-#endif
- else if (PyObject_AsCharBuffer(substr, &sub, &slen))
- return -1;
- str = PyBytes_AS_STRING(self);
-
- string_adjust_indices(&start, &end, len);
-
- if (direction < 0) {
- /* startswith */
- if (start+slen > len)
- return 0;
- } else {
- /* endswith */
- if (end-start < slen || start > len)
- return 0;
-
- if (end-slen > start)
- start = end - slen;
- }
- if (end-start >= slen)
- return ! memcmp(str+start, sub, slen);
- return 0;
-}
-
-
-PyDoc_STRVAR(startswith__doc__,
-"S.startswith(prefix[, start[, end]]) -> bool\n\
-\n\
-Return True if S starts with the specified prefix, False otherwise.\n\
-With optional start, test S beginning at that position.\n\
-With optional end, stop comparing S at that position.\n\
-prefix can also be a tuple of strings to try.");
-
-static PyObject *
-string_startswith(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t start = 0;
- Py_ssize_t end = PY_SSIZE_T_MAX;
- PyObject *subobj;
- int result;
-
- if (!PyArg_ParseTuple(args, "O|O&O&:startswith", &subobj,
- _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end))
- return NULL;
- if (PyTuple_Check(subobj)) {
- Py_ssize_t i;
- for (i = 0; i < PyTuple_GET_SIZE(subobj); i++) {
- result = _string_tailmatch(self,
- PyTuple_GET_ITEM(subobj, i),
- start, end, -1);
- if (result == -1)
- return NULL;
- else if (result) {
- Py_RETURN_TRUE;
- }
- }
- Py_RETURN_FALSE;
- }
- result = _string_tailmatch(self, subobj, start, end, -1);
- if (result == -1)
- return NULL;
- else
- return PyBool_FromLong(result);
-}
-
-
-PyDoc_STRVAR(endswith__doc__,
-"S.endswith(suffix[, start[, end]]) -> bool\n\
-\n\
-Return True if S ends with the specified suffix, False otherwise.\n\
-With optional start, test S beginning at that position.\n\
-With optional end, stop comparing S at that position.\n\
-suffix can also be a tuple of strings to try.");
-
-static PyObject *
-string_endswith(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t start = 0;
- Py_ssize_t end = PY_SSIZE_T_MAX;
- PyObject *subobj;
- int result;
-
- if (!PyArg_ParseTuple(args, "O|O&O&:endswith", &subobj,
- _PyEval_SliceIndex, &start, _PyEval_SliceIndex, &end))
- return NULL;
- if (PyTuple_Check(subobj)) {
- Py_ssize_t i;
- for (i = 0; i < PyTuple_GET_SIZE(subobj); i++) {
- result = _string_tailmatch(self,
- PyTuple_GET_ITEM(subobj, i),
- start, end, +1);
- if (result == -1)
- return NULL;
- else if (result) {
- Py_RETURN_TRUE;
- }
- }
- Py_RETURN_FALSE;
- }
- result = _string_tailmatch(self, subobj, start, end, +1);
- if (result == -1)
- return NULL;
- else
- return PyBool_FromLong(result);
-}
-
-
-PyDoc_STRVAR(encode__doc__,
-"S.encode([encoding[,errors]]) -> object\n\
-\n\
-Encodes S using the codec registered for encoding. encoding defaults\n\
-to the default encoding. errors may be given to set a different error\n\
-handling scheme. Default is 'strict' meaning that encoding errors raise\n\
-a UnicodeEncodeError. Other possible values are 'ignore', 'replace' and\n\
-'xmlcharrefreplace' as well as any other name registered with\n\
-codecs.register_error that is able to handle UnicodeEncodeErrors.");
-
-static PyObject *
-string_encode(PyBytesObject *self, PyObject *args)
-{
- char *encoding = NULL;
- char *errors = NULL;
- PyObject *v;
-
- if (!PyArg_ParseTuple(args, "|ss:encode", &encoding, &errors))
- return NULL;
- v = PyBytes_AsEncodedObject((PyObject *)self, encoding, errors);
- if (v == NULL)
- goto onError;
- if (!PyBytes_Check(v) && !PyUnicode_Check(v)) {
- PyErr_Format(PyExc_TypeError,
- "encoder did not return a string/unicode object "
- "(type=%.400s)",
- Py_TYPE(v)->tp_name);
- Py_DECREF(v);
- return NULL;
- }
- return v;
-
- onError:
- return NULL;
-}
-
-
-PyDoc_STRVAR(decode__doc__,
-"S.decode([encoding[,errors]]) -> object\n\
-\n\
-Decodes S using the codec registered for encoding. encoding defaults\n\
-to the default encoding. errors may be given to set a different error\n\
-handling scheme. Default is 'strict' meaning that encoding errors raise\n\
-a UnicodeDecodeError. Other possible values are 'ignore' and 'replace'\n\
-as well as any other name registerd with codecs.register_error that is\n\
-able to handle UnicodeDecodeErrors.");
-
-static PyObject *
-string_decode(PyBytesObject *self, PyObject *args)
-{
- char *encoding = NULL;
- char *errors = NULL;
- PyObject *v;
-
- if (!PyArg_ParseTuple(args, "|ss:decode", &encoding, &errors))
- return NULL;
- v = PyBytes_AsDecodedObject((PyObject *)self, encoding, errors);
- if (v == NULL)
- goto onError;
- if (!PyBytes_Check(v) && !PyUnicode_Check(v)) {
- PyErr_Format(PyExc_TypeError,
- "decoder did not return a string/unicode object "
- "(type=%.400s)",
- Py_TYPE(v)->tp_name);
- Py_DECREF(v);
- return NULL;
- }
- return v;
-
- onError:
- return NULL;
-}
-
-
-PyDoc_STRVAR(expandtabs__doc__,
-"S.expandtabs([tabsize]) -> string\n\
-\n\
-Return a copy of S where all tab characters are expanded using spaces.\n\
-If tabsize is not given, a tab size of 8 characters is assumed.");
-
-static PyObject*
-string_expandtabs(PyBytesObject *self, PyObject *args)
-{
- const char *e, *p, *qe;
- char *q;
- Py_ssize_t i, j, incr;
- PyObject *u;
- int tabsize = 8;
-
- if (!PyArg_ParseTuple(args, "|i:expandtabs", &tabsize))
- return NULL;
-
- /* First pass: determine size of output string */
- i = 0; /* chars up to and including most recent \n or \r */
- j = 0; /* chars since most recent \n or \r (use in tab calculations) */
- e = PyBytes_AS_STRING(self) + PyBytes_GET_SIZE(self); /* end of input */
- for (p = PyBytes_AS_STRING(self); p < e; p++)
- if (*p == '\t') {
- if (tabsize > 0) {
- incr = tabsize - (j % tabsize);
- if (j > PY_SSIZE_T_MAX - incr)
- goto overflow1;
- j += incr;
- }
- }
- else {
- if (j > PY_SSIZE_T_MAX - 1)
- goto overflow1;
- j++;
- if (*p == '\n' || *p == '\r') {
- if (i > PY_SSIZE_T_MAX - j)
- goto overflow1;
- i += j;
- j = 0;
- }
- }
-
- if (i > PY_SSIZE_T_MAX - j)
- goto overflow1;
-
- /* Second pass: create output string and fill it */
- u = PyBytes_FromStringAndSize(NULL, i + j);
- if (!u)
- return NULL;
-
- j = 0; /* same as in first pass */
- q = PyBytes_AS_STRING(u); /* next output char */
- qe = PyBytes_AS_STRING(u) + PyBytes_GET_SIZE(u); /* end of output */
-
- for (p = PyBytes_AS_STRING(self); p < e; p++)
- if (*p == '\t') {
- if (tabsize > 0) {
- i = tabsize - (j % tabsize);
- j += i;
- while (i--) {
- if (q >= qe)
- goto overflow2;
- *q++ = ' ';
- }
- }
- }
- else {
- if (q >= qe)
- goto overflow2;
- *q++ = *p;
- j++;
- if (*p == '\n' || *p == '\r')
- j = 0;
- }
-
- return u;
-
- overflow2:
- Py_DECREF(u);
- overflow1:
- PyErr_SetString(PyExc_OverflowError, "new string is too long");
- return NULL;
-}
-
-Py_LOCAL_INLINE(PyObject *)
-pad(PyBytesObject *self, Py_ssize_t left, Py_ssize_t right, char fill)
-{
- PyObject *u;
-
- if (left < 0)
- left = 0;
- if (right < 0)
- right = 0;
-
- if (left == 0 && right == 0 && PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject *)self;
- }
-
- u = PyBytes_FromStringAndSize(NULL,
- left + PyBytes_GET_SIZE(self) + right);
- if (u) {
- if (left)
- memset(PyBytes_AS_STRING(u), fill, left);
- Py_MEMCPY(PyBytes_AS_STRING(u) + left,
- PyBytes_AS_STRING(self),
- PyBytes_GET_SIZE(self));
- if (right)
- memset(PyBytes_AS_STRING(u) + left + PyBytes_GET_SIZE(self),
- fill, right);
- }
-
- return u;
-}
-
-PyDoc_STRVAR(ljust__doc__,
-"S.ljust(width[, fillchar]) -> string\n"
-"\n"
-"Return S left justified in a string of length width. Padding is\n"
-"done using the specified fill character (default is a space).");
-
-static PyObject *
-string_ljust(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t width;
- char fillchar = ' ';
-
- if (!PyArg_ParseTuple(args, "n|c:ljust", &width, &fillchar))
- return NULL;
-
- if (PyBytes_GET_SIZE(self) >= width && PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject*) self;
- }
-
- return pad(self, 0, width - PyBytes_GET_SIZE(self), fillchar);
-}
-
-
-PyDoc_STRVAR(rjust__doc__,
-"S.rjust(width[, fillchar]) -> string\n"
-"\n"
-"Return S right justified in a string of length width. Padding is\n"
-"done using the specified fill character (default is a space)");
-
-static PyObject *
-string_rjust(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t width;
- char fillchar = ' ';
-
- if (!PyArg_ParseTuple(args, "n|c:rjust", &width, &fillchar))
- return NULL;
-
- if (PyBytes_GET_SIZE(self) >= width && PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject*) self;
- }
-
- return pad(self, width - PyBytes_GET_SIZE(self), 0, fillchar);
-}
-
-
-PyDoc_STRVAR(center__doc__,
-"S.center(width[, fillchar]) -> string\n"
-"\n"
-"Return S centered in a string of length width. Padding is\n"
-"done using the specified fill character (default is a space)");
-
-static PyObject *
-string_center(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t marg, left;
- Py_ssize_t width;
- char fillchar = ' ';
-
- if (!PyArg_ParseTuple(args, "n|c:center", &width, &fillchar))
- return NULL;
-
- if (PyBytes_GET_SIZE(self) >= width && PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject*) self;
- }
-
- marg = width - PyBytes_GET_SIZE(self);
- left = marg / 2 + (marg & width & 1);
-
- return pad(self, left, marg - left, fillchar);
-}
-
-PyDoc_STRVAR(zfill__doc__,
-"S.zfill(width) -> string\n"
-"\n"
-"Pad a numeric string S with zeros on the left, to fill a field\n"
-"of the specified width. The string S is never truncated.");
-
-static PyObject *
-string_zfill(PyBytesObject *self, PyObject *args)
-{
- Py_ssize_t fill;
- PyObject *s;
- char *p;
- Py_ssize_t width;
-
- if (!PyArg_ParseTuple(args, "n:zfill", &width))
- return NULL;
-
- if (PyBytes_GET_SIZE(self) >= width) {
- if (PyBytes_CheckExact(self)) {
- Py_INCREF(self);
- return (PyObject*) self;
- }
- else
- return PyBytes_FromStringAndSize(
- PyBytes_AS_STRING(self),
- PyBytes_GET_SIZE(self)
- );
- }
-
- fill = width - PyBytes_GET_SIZE(self);
-
- s = pad(self, fill, 0, '0');
-
- if (s == NULL)
- return NULL;
-
- p = PyBytes_AS_STRING(s);
- if (p[fill] == '+' || p[fill] == '-') {
- /* move sign to beginning of string */
- p[0] = p[fill];
- p[fill] = '0';
- }
-
- return (PyObject*) s;
-}
-
-PyDoc_STRVAR(isspace__doc__,
-"S.isspace() -> bool\n\
-\n\
-Return True if all characters in S are whitespace\n\
-and there is at least one character in S, False otherwise.");
-
-static PyObject*
-string_isspace(PyBytesObject *self)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1 &&
- isspace(*p))
- return PyBool_FromLong(1);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- for (; p < e; p++) {
- if (!isspace(*p))
- return PyBool_FromLong(0);
- }
- return PyBool_FromLong(1);
-}
-
-
-PyDoc_STRVAR(isalpha__doc__,
-"S.isalpha() -> bool\n\
-\n\
-Return True if all characters in S are alphabetic\n\
-and there is at least one character in S, False otherwise.");
-
-static PyObject*
-string_isalpha(PyBytesObject *self)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1 &&
- isalpha(*p))
- return PyBool_FromLong(1);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- for (; p < e; p++) {
- if (!isalpha(*p))
- return PyBool_FromLong(0);
- }
- return PyBool_FromLong(1);
-}
-
-
-PyDoc_STRVAR(isalnum__doc__,
-"S.isalnum() -> bool\n\
-\n\
-Return True if all characters in S are alphanumeric\n\
-and there is at least one character in S, False otherwise.");
-
-static PyObject*
-string_isalnum(PyBytesObject *self)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1 &&
- isalnum(*p))
- return PyBool_FromLong(1);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- for (; p < e; p++) {
- if (!isalnum(*p))
- return PyBool_FromLong(0);
- }
- return PyBool_FromLong(1);
-}
-
-
-PyDoc_STRVAR(isdigit__doc__,
-"S.isdigit() -> bool\n\
-\n\
-Return True if all characters in S are digits\n\
-and there is at least one character in S, False otherwise.");
-
-static PyObject*
-string_isdigit(PyBytesObject *self)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1 &&
- isdigit(*p))
- return PyBool_FromLong(1);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- for (; p < e; p++) {
- if (!isdigit(*p))
- return PyBool_FromLong(0);
- }
- return PyBool_FromLong(1);
-}
-
-
-PyDoc_STRVAR(islower__doc__,
-"S.islower() -> bool\n\
-\n\
-Return True if all cased characters in S are lowercase and there is\n\
-at least one cased character in S, False otherwise.");
-
-static PyObject*
-string_islower(PyBytesObject *self)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
- int cased;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1)
- return PyBool_FromLong(islower(*p) != 0);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- cased = 0;
- for (; p < e; p++) {
- if (isupper(*p))
- return PyBool_FromLong(0);
- else if (!cased && islower(*p))
- cased = 1;
- }
- return PyBool_FromLong(cased);
-}
-
-
-PyDoc_STRVAR(isupper__doc__,
-"S.isupper() -> bool\n\
-\n\
-Return True if all cased characters in S are uppercase and there is\n\
-at least one cased character in S, False otherwise.");
-
-static PyObject*
-string_isupper(PyBytesObject *self)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
- int cased;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1)
- return PyBool_FromLong(isupper(*p) != 0);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- cased = 0;
- for (; p < e; p++) {
- if (islower(*p))
- return PyBool_FromLong(0);
- else if (!cased && isupper(*p))
- cased = 1;
- }
- return PyBool_FromLong(cased);
-}
-
-
-PyDoc_STRVAR(istitle__doc__,
-"S.istitle() -> bool\n\
-\n\
-Return True if S is a titlecased string and there is at least one\n\
-character in S, i.e. uppercase characters may only follow uncased\n\
-characters and lowercase characters only cased ones. Return False\n\
-otherwise.");
-
-static PyObject*
-string_istitle(PyBytesObject *self, PyObject *uncased)
-{
- register const unsigned char *p
- = (unsigned char *) PyBytes_AS_STRING(self);
- register const unsigned char *e;
- int cased, previous_is_cased;
-
- /* Shortcut for single character strings */
- if (PyBytes_GET_SIZE(self) == 1)
- return PyBool_FromLong(isupper(*p) != 0);
-
- /* Special case for empty strings */
- if (PyBytes_GET_SIZE(self) == 0)
- return PyBool_FromLong(0);
-
- e = p + PyBytes_GET_SIZE(self);
- cased = 0;
- previous_is_cased = 0;
- for (; p < e; p++) {
- register const unsigned char ch = *p;
-
- if (isupper(ch)) {
- if (previous_is_cased)
- return PyBool_FromLong(0);
- previous_is_cased = 1;
- cased = 1;
- }
- else if (islower(ch)) {
- if (!previous_is_cased)
- return PyBool_FromLong(0);
- previous_is_cased = 1;
- cased = 1;
- }
- else
- previous_is_cased = 0;
- }
- return PyBool_FromLong(cased);
-}
-
-
-PyDoc_STRVAR(splitlines__doc__,
-"S.splitlines([keepends]) -> list of strings\n\
-\n\
-Return a list of the lines in S, breaking at line boundaries.\n\
-Line breaks are not included in the resulting list unless keepends\n\
-is given and true.");
-
-static PyObject*
-string_splitlines(PyBytesObject *self, PyObject *args)
-{
- register Py_ssize_t i;
- register Py_ssize_t j;
- Py_ssize_t len;
- int keepends = 0;
- PyObject *list;
- PyObject *str;
- char *data;
-
- if (!PyArg_ParseTuple(args, "|i:splitlines", &keepends))
- return NULL;
-
- data = PyBytes_AS_STRING(self);
- len = PyBytes_GET_SIZE(self);
-
- /* This does not use the preallocated list because splitlines is
- usually run with hundreds of newlines. The overhead of
- switching between PyList_SET_ITEM and append causes about a
- 2-3% slowdown for that common case. A smarter implementation
- could move the if check out, so the SET_ITEMs are done first
- and the appends only done when the prealloc buffer is full.
- That's too much work for little gain.*/
-
- list = PyList_New(0);
- if (!list)
- goto onError;
-
- for (i = j = 0; i < len; ) {
- Py_ssize_t eol;
-
- /* Find a line and append it */
- while (i < len && data[i] != '\n' && data[i] != '\r')
- i++;
-
- /* Skip the line break reading CRLF as one line break */
- eol = i;
- if (i < len) {
- if (data[i] == '\r' && i + 1 < len &&
- data[i+1] == '\n')
- i += 2;
- else
- i++;
- if (keepends)
- eol = i;
- }
- SPLIT_APPEND(data, j, eol);
- j = i;
- }
- if (j < len) {
- SPLIT_APPEND(data, j, len);
- }
-
- return list;
-
- onError:
- Py_XDECREF(list);
- return NULL;
-}
-
-PyDoc_STRVAR(sizeof__doc__,
-"S.__sizeof__() -> size of S in memory, in bytes");
-
-static PyObject *
-string_sizeof(PyBytesObject *v)
-{
- Py_ssize_t res;
- res = sizeof(PyBytesObject) + v->ob_size * v->ob_type->tp_itemsize;
- return PyInt_FromSsize_t(res);
-}
-
-#undef SPLIT_APPEND
-#undef SPLIT_ADD
-#undef MAX_PREALLOC
-#undef PREALLOC_SIZE
-
-static PyObject *
-string_getnewargs(PyBytesObject *v)
-{
- return Py_BuildValue("(s#)", v->ob_sval, Py_SIZE(v));
-}
-
-
-#include "stringlib/string_format.h"
-
-PyDoc_STRVAR(format__doc__,
-"S.format(*args, **kwargs) -> unicode\n\
-\n\
-");
-
-static PyObject *
-string__format__(PyObject* self, PyObject* args)
-{
- PyObject *format_spec;
- PyObject *result = NULL;
- PyObject *tmp = NULL;
-
- /* If 2.x, convert format_spec to the same type as value */
- /* This is to allow things like u''.format('') */
- if (!PyArg_ParseTuple(args, "O:__format__", &format_spec))
- goto done;
- if (!(PyBytes_Check(format_spec) || PyUnicode_Check(format_spec))) {
- PyErr_Format(PyExc_TypeError, "__format__ arg must be str "
- "or unicode, not %s", Py_TYPE(format_spec)->tp_name);
- goto done;
- }
- tmp = PyObject_Str(format_spec);
- if (tmp == NULL)
- goto done;
- format_spec = tmp;
-
- result = _PyBytes_FormatAdvanced(self,
- PyBytes_AS_STRING(format_spec),
- PyBytes_GET_SIZE(format_spec));
-done:
- Py_XDECREF(tmp);
- return result;
-}
-
-PyDoc_STRVAR(p_format__doc__,
-"S.__format__(format_spec) -> unicode\n\
-\n\
-");
-
-
-static PyMethodDef
-string_methods[] = {
- /* Counterparts of the obsolete stropmodule functions; except
- string.maketrans(). */
- {"join", (PyCFunction)string_join, METH_O, join__doc__},
- {"split", (PyCFunction)string_split, METH_VARARGS, split__doc__},
- {"rsplit", (PyCFunction)string_rsplit, METH_VARARGS, rsplit__doc__},
- {"lower", (PyCFunction)string_lower, METH_NOARGS, lower__doc__},
- {"upper", (PyCFunction)string_upper, METH_NOARGS, upper__doc__},
- {"islower", (PyCFunction)string_islower, METH_NOARGS, islower__doc__},
- {"isupper", (PyCFunction)string_isupper, METH_NOARGS, isupper__doc__},
- {"isspace", (PyCFunction)string_isspace, METH_NOARGS, isspace__doc__},
- {"isdigit", (PyCFunction)string_isdigit, METH_NOARGS, isdigit__doc__},
- {"istitle", (PyCFunction)string_istitle, METH_NOARGS, istitle__doc__},
- {"isalpha", (PyCFunction)string_isalpha, METH_NOARGS, isalpha__doc__},
- {"isalnum", (PyCFunction)string_isalnum, METH_NOARGS, isalnum__doc__},
- {"capitalize", (PyCFunction)string_capitalize, METH_NOARGS,
- capitalize__doc__},
- {"count", (PyCFunction)string_count, METH_VARARGS, count__doc__},
- {"endswith", (PyCFunction)string_endswith, METH_VARARGS,
- endswith__doc__},
- {"partition", (PyCFunction)string_partition, METH_O, partition__doc__},
- {"find", (PyCFunction)string_find, METH_VARARGS, find__doc__},
- {"index", (PyCFunction)string_index, METH_VARARGS, index__doc__},
- {"lstrip", (PyCFunction)string_lstrip, METH_VARARGS, lstrip__doc__},
- {"replace", (PyCFunction)string_replace, METH_VARARGS, replace__doc__},
- {"rfind", (PyCFunction)string_rfind, METH_VARARGS, rfind__doc__},
- {"rindex", (PyCFunction)string_rindex, METH_VARARGS, rindex__doc__},
- {"rstrip", (PyCFunction)string_rstrip, METH_VARARGS, rstrip__doc__},
- {"rpartition", (PyCFunction)string_rpartition, METH_O,
- rpartition__doc__},
- {"startswith", (PyCFunction)string_startswith, METH_VARARGS,
- startswith__doc__},
- {"strip", (PyCFunction)string_strip, METH_VARARGS, strip__doc__},
- {"swapcase", (PyCFunction)string_swapcase, METH_NOARGS,
- swapcase__doc__},
- {"translate", (PyCFunction)string_translate, METH_VARARGS,
- translate__doc__},
- {"title", (PyCFunction)string_title, METH_NOARGS, title__doc__},
- {"ljust", (PyCFunction)string_ljust, METH_VARARGS, ljust__doc__},
- {"rjust", (PyCFunction)string_rjust, METH_VARARGS, rjust__doc__},
- {"center", (PyCFunction)string_center, METH_VARARGS, center__doc__},
- {"zfill", (PyCFunction)string_zfill, METH_VARARGS, zfill__doc__},
- {"format", (PyCFunction) do_string_format, METH_VARARGS | METH_KEYWORDS, format__doc__},
- {"__format__", (PyCFunction) string__format__, METH_VARARGS, p_format__doc__},
- {"_formatter_field_name_split", (PyCFunction) formatter_field_name_split, METH_NOARGS},
- {"_formatter_parser", (PyCFunction) formatter_parser, METH_NOARGS},
- {"encode", (PyCFunction)string_encode, METH_VARARGS, encode__doc__},
- {"decode", (PyCFunction)string_decode, METH_VARARGS, decode__doc__},
- {"expandtabs", (PyCFunction)string_expandtabs, METH_VARARGS,
- expandtabs__doc__},
- {"splitlines", (PyCFunction)string_splitlines, METH_VARARGS,
- splitlines__doc__},
- {"__sizeof__", (PyCFunction)string_sizeof, METH_NOARGS,
- sizeof__doc__},
- {"__getnewargs__", (PyCFunction)string_getnewargs, METH_NOARGS},
- {NULL, NULL} /* sentinel */
-};
-
-static PyObject *
-str_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds);
-
-static PyObject *
-string_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
-{
- PyObject *x = NULL;
- static char *kwlist[] = {"object", 0};
-
- if (type != &PyBytes_Type)
- return str_subtype_new(type, args, kwds);
- if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O:str", kwlist, &x))
- return NULL;
- if (x == NULL)
- return PyBytes_FromString("");
- return PyObject_Str(x);
-}
-
-static PyObject *
-str_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
-{
- PyObject *tmp, *pnew;
- Py_ssize_t n;
-
- assert(PyType_IsSubtype(type, &PyBytes_Type));
- tmp = string_new(&PyBytes_Type, args, kwds);
- if (tmp == NULL)
- return NULL;
- assert(PyBytes_CheckExact(tmp));
- n = PyBytes_GET_SIZE(tmp);
- pnew = type->tp_alloc(type, n);
- if (pnew != NULL) {
- Py_MEMCPY(PyBytes_AS_STRING(pnew), PyBytes_AS_STRING(tmp), n+1);
- ((PyBytesObject *)pnew)->ob_shash =
- ((PyBytesObject *)tmp)->ob_shash;
- ((PyBytesObject *)pnew)->ob_sstate = SSTATE_NOT_INTERNED;
- }
- Py_DECREF(tmp);
- return pnew;
-}
-
-static PyObject *
-basestring_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
-{
- PyErr_SetString(PyExc_TypeError,
- "The basestring type cannot be instantiated");
- return NULL;
-}
-
-static PyObject *
-string_mod(PyObject *v, PyObject *w)
-{
- if (!PyBytes_Check(v)) {
- Py_INCREF(Py_NotImplemented);
- return Py_NotImplemented;
- }
- return PyBytes_Format(v, w);
-}
-
-PyDoc_STRVAR(basestring_doc,
-"Type basestring cannot be instantiated; it is the base for str and unicode.");
-
-static PyNumberMethods string_as_number = {
- 0, /*nb_add*/
- 0, /*nb_subtract*/
- 0, /*nb_multiply*/
- 0, /*nb_divide*/
- string_mod, /*nb_remainder*/
-};
-
-
-PyTypeObject PyBaseString_Type = {
- PyVarObject_HEAD_INIT(&PyType_Type, 0)
- "basestring",
- 0,
- 0,
- 0, /* tp_dealloc */
- 0, /* tp_print */
- 0, /* tp_getattr */
- 0, /* tp_setattr */
- 0, /* tp_compare */
- 0, /* tp_repr */
- 0, /* tp_as_number */
- 0, /* tp_as_sequence */
- 0, /* tp_as_mapping */
- 0, /* tp_hash */
- 0, /* tp_call */
- 0, /* tp_str */
- 0, /* tp_getattro */
- 0, /* tp_setattro */
- 0, /* tp_as_buffer */
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
- basestring_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
- 0, /* tp_richcompare */
- 0, /* tp_weaklistoffset */
- 0, /* tp_iter */
- 0, /* tp_iternext */
- 0, /* tp_methods */
- 0, /* tp_members */
- 0, /* tp_getset */
- &PyBaseObject_Type, /* tp_base */
- 0, /* tp_dict */
- 0, /* tp_descr_get */
- 0, /* tp_descr_set */
- 0, /* tp_dictoffset */
- 0, /* tp_init */
- 0, /* tp_alloc */
- basestring_new, /* tp_new */
- 0, /* tp_free */
-};
-
-PyDoc_STRVAR(string_doc,
-"str(object) -> string\n\
-\n\
-Return a nice string representation of the object.\n\
-If the argument is a string, the return value is the same object.");
-
-PyTypeObject PyBytes_Type = {
- PyVarObject_HEAD_INIT(&PyType_Type, 0)
- "str",
- sizeof(PyBytesObject),
- sizeof(char),
- string_dealloc, /* tp_dealloc */
- (printfunc)string_print, /* tp_print */
- 0, /* tp_getattr */
- 0, /* tp_setattr */
- 0, /* tp_compare */
- string_repr, /* tp_repr */
- &string_as_number, /* tp_as_number */
- &string_as_sequence, /* tp_as_sequence */
- &string_as_mapping, /* tp_as_mapping */
- (hashfunc)string_hash, /* tp_hash */
- 0, /* tp_call */
- string_str, /* tp_str */
- PyObject_GenericGetAttr, /* tp_getattro */
- 0, /* tp_setattro */
- &string_as_buffer, /* tp_as_buffer */
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_CHECKTYPES |
- Py_TPFLAGS_BASETYPE | Py_TPFLAGS_STRING_SUBCLASS |
- Py_TPFLAGS_HAVE_NEWBUFFER, /* tp_flags */
- string_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
- (richcmpfunc)string_richcompare, /* tp_richcompare */
- 0, /* tp_weaklistoffset */
- 0, /* tp_iter */
- 0, /* tp_iternext */
- string_methods, /* tp_methods */
- 0, /* tp_members */
- 0, /* tp_getset */
- &PyBaseString_Type, /* tp_base */
- 0, /* tp_dict */
- 0, /* tp_descr_get */
- 0, /* tp_descr_set */
- 0, /* tp_dictoffset */
- 0, /* tp_init */
- 0, /* tp_alloc */
- string_new, /* tp_new */
- PyObject_Del, /* tp_free */
-};
-
-void
-PyBytes_Concat(register PyObject **pv, register PyObject *w)
-{
- register PyObject *v;
- if (*pv == NULL)
- return;
- if (w == NULL || !PyBytes_Check(*pv)) {
- Py_DECREF(*pv);
- *pv = NULL;
- return;
- }
- v = string_concat((PyBytesObject *) *pv, w);
- Py_DECREF(*pv);
- *pv = v;
-}
-
-void
-PyBytes_ConcatAndDel(register PyObject **pv, register PyObject *w)
-{
- PyBytes_Concat(pv, w);
- Py_XDECREF(w);
-}
-
-
-/* The following function breaks the notion that strings are immutable:
- it changes the size of a string. We get away with this only if there
- is only one module referencing the object. You can also think of it
- as creating a new string object and destroying the old one, only
- more efficiently. In any case, don't use this if the string may
- already be known to some other part of the code...
- Note that if there's not enough memory to resize the string, the original
- string object at *pv is deallocated, *pv is set to NULL, an "out of
- memory" exception is set, and -1 is returned. Else (on success) 0 is
- returned, and the value in *pv may or may not be the same as on input.
- As always, an extra byte is allocated for a trailing \0 byte (newsize
- does *not* include that), and a trailing \0 byte is stored.
-*/
-
-int
-_PyBytes_Resize(PyObject **pv, Py_ssize_t newsize)
-{
- register PyObject *v;
- register PyBytesObject *sv;
- v = *pv;
- if (!PyBytes_Check(v) || Py_REFCNT(v) != 1 || newsize < 0 ||
- PyBytes_CHECK_INTERNED(v)) {
- *pv = 0;
- Py_DECREF(v);
- PyErr_BadInternalCall();
- return -1;
- }
- /* XXX UNREF/NEWREF interface should be more symmetrical */
- _Py_DEC_REFTOTAL;
- _Py_ForgetReference(v);
- *pv = (PyObject *)
- PyObject_REALLOC((char *)v, sizeof(PyBytesObject) + newsize);
- if (*pv == NULL) {
- PyObject_Del(v);
- PyErr_NoMemory();
- return -1;
- }
- _Py_NewReference(*pv);
- sv = (PyBytesObject *) *pv;
- Py_SIZE(sv) = newsize;
- sv->ob_sval[newsize] = '\0';
- sv->ob_shash = -1; /* invalidate cached hash value */
- return 0;
-}
-
-/* Helpers for formatstring */
-
-Py_LOCAL_INLINE(PyObject *)
-getnextarg(PyObject *args, Py_ssize_t arglen, Py_ssize_t *p_argidx)
-{
- Py_ssize_t argidx = *p_argidx;
- if (argidx < arglen) {
- (*p_argidx)++;
- if (arglen < 0)
- return args;
- else
- return PyTuple_GetItem(args, argidx);
- }
- PyErr_SetString(PyExc_TypeError,
- "not enough arguments for format string");
- return NULL;
-}
-
-/* Format codes
- * F_LJUST '-'
- * F_SIGN '+'
- * F_BLANK ' '
- * F_ALT '#'
- * F_ZERO '0'
- */
-#define F_LJUST (1<<0)
-#define F_SIGN (1<<1)
-#define F_BLANK (1<<2)
-#define F_ALT (1<<3)
-#define F_ZERO (1<<4)
-
-Py_LOCAL_INLINE(int)
-formatfloat(char *buf, size_t buflen, int flags,
- int prec, int type, PyObject *v)
-{
- /* fmt = '%#.' + `prec` + `type`
- worst case length = 3 + 10 (len of INT_MAX) + 1 = 14 (use 20)*/
- char fmt[20];
- double x;
- x = PyFloat_AsDouble(v);
- if (x == -1.0 && PyErr_Occurred()) {
- PyErr_Format(PyExc_TypeError, "float argument required, "
- "not %.200s", Py_TYPE(v)->tp_name);
- return -1;
- }
- if (prec < 0)
- prec = 6;
- if (type == 'f' && fabs(x)/1e25 >= 1e25)
- type = 'g';
- /* Worst case length calc to ensure no buffer overrun:
-
- 'g' formats:
- fmt = %#.<prec>g
- buf = '-' + [0-9]*prec + '.' + 'e+' + (longest exp
- for any double rep.)
- len = 1 + prec + 1 + 2 + 5 = 9 + prec
-
- 'f' formats:
- buf = '-' + [0-9]*x + '.' + [0-9]*prec (with x < 50)
- len = 1 + 50 + 1 + prec = 52 + prec
-
- If prec=0 the effective precision is 1 (the leading digit is
- always given), therefore increase the length by one.
-
- */
- if (((type == 'g' || type == 'G') &&
- buflen <= (size_t)10 + (size_t)prec) ||
- (type == 'f' && buflen <= (size_t)53 + (size_t)prec)) {
- PyErr_SetString(PyExc_OverflowError,
- "formatted float is too long (precision too large?)");
- return -1;
- }
- PyOS_snprintf(fmt, sizeof(fmt), "%%%s.%d%c",
- (flags&F_ALT) ? "#" : "",
- prec, type);
- PyOS_ascii_formatd(buf, buflen, fmt, x);
- return (int)strlen(buf);
-}
-
-/* _PyBytes_FormatLong emulates the format codes d, u, o, x and X, and
- * the F_ALT flag, for Python's long (unbounded) ints. It's not used for
- * Python's regular ints.
- * Return value: a new PyString*, or NULL if error.
- * . *pbuf is set to point into it,
- * *plen set to the # of chars following that.
- * Caller must decref it when done using pbuf.
- * The string starting at *pbuf is of the form
- * "-"? ("0x" | "0X")? digit+
- * "0x"/"0X" are present only for x and X conversions, with F_ALT
- * set in flags. The case of hex digits will be correct,
- * There will be at least prec digits, zero-filled on the left if
- * necessary to get that many.
- * val object to be converted
- * flags bitmask of format flags; only F_ALT is looked at
- * prec minimum number of digits; 0-fill on left if needed
- * type a character in [duoxX]; u acts the same as d
- *
- * CAUTION: o, x and X conversions on regular ints can never
- * produce a '-' sign, but can for Python's unbounded ints.
- */
-PyObject*
-_PyBytes_FormatLong(PyObject *val, int flags, int prec, int type,
- char **pbuf, int *plen)
-{
- PyObject *result = NULL;
- char *buf;
- Py_ssize_t i;
- int sign; /* 1 if '-', else 0 */
- int len; /* number of characters */
- Py_ssize_t llen;
- int numdigits; /* len == numnondigits + numdigits */
- int numnondigits = 0;
-
- switch (type) {
- case 'd':
- case 'u':
- result = Py_TYPE(val)->tp_str(val);
- break;
- case 'o':
- result = Py_TYPE(val)->tp_as_number->nb_oct(val);
- break;
- case 'x':
- case 'X':
- numnondigits = 2;
- result = Py_TYPE(val)->tp_as_number->nb_hex(val);
- break;
- default:
- assert(!"'type' not in [duoxX]");
- }
- if (!result)
- return NULL;
-
- buf = PyBytes_AsString(result);
- if (!buf) {
- Py_DECREF(result);
- return NULL;
- }
-
- /* To modify the string in-place, there can only be one reference. */
- if (Py_REFCNT(result) != 1) {
- PyErr_BadInternalCall();
- return NULL;
- }
- llen = PyBytes_Size(result);
- if (llen > INT_MAX) {
- PyErr_SetString(PyExc_ValueError, "string too large in _PyBytes_FormatLong");
- return NULL;
- }
- len = (int)llen;
- if (buf[len-1] == 'L') {
- --len;
- buf[len] = '\0';
- }
- sign = buf[0] == '-';
- numnondigits += sign;
- numdigits = len - numnondigits;
- assert(numdigits > 0);
-
- /* Get rid of base marker unless F_ALT */
- if ((flags & F_ALT) == 0) {
- /* Need to skip 0x, 0X or 0. */
- int skipped = 0;
- switch (type) {
- case 'o':
- assert(buf[sign] == '0');
- /* If 0 is only digit, leave it alone. */
- if (numdigits > 1) {
- skipped = 1;
- --numdigits;
- }
- break;
- case 'x':
- case 'X':
- assert(buf[sign] == '0');
- assert(buf[sign + 1] == 'x');
- skipped = 2;
- numnondigits -= 2;
- break;
- }
- if (skipped) {
- buf += skipped;
- len -= skipped;
- if (sign)
- buf[0] = '-';
- }
- assert(len == numnondigits + numdigits);
- assert(numdigits > 0);
- }
-
- /* Fill with leading zeroes to meet minimum width. */
- if (prec > numdigits) {
- PyObject *r1 = PyBytes_FromStringAndSize(NULL,
- numnondigits + prec);
- char *b1;
- if (!r1) {
- Py_DECREF(result);
- return NULL;
- }
- b1 = PyBytes_AS_STRING(r1);
- for (i = 0; i < numnondigits; ++i)
- *b1++ = *buf++;
- for (i = 0; i < prec - numdigits; i++)
- *b1++ = '0';
- for (i = 0; i < numdigits; i++)
- *b1++ = *buf++;
- *b1 = '\0';
- Py_DECREF(result);
- result = r1;
- buf = PyBytes_AS_STRING(result);
- len = numnondigits + prec;
- }
-
- /* Fix up case for hex conversions. */
- if (type == 'X') {
- /* Need to convert all lower case letters to upper case.
- and need to convert 0x to 0X (and -0x to -0X). */
- for (i = 0; i < len; i++)
- if (buf[i] >= 'a' && buf[i] <= 'x')
- buf[i] -= 'a'-'A';
- }
- *pbuf = buf;
- *plen = len;
- return result;
-}
-
-Py_LOCAL_INLINE(int)
-formatint(char *buf, size_t buflen, int flags,
- int prec, int type, PyObject *v)
-{
- /* fmt = '%#.' + `prec` + 'l' + `type`
- worst case length = 3 + 19 (worst len of INT_MAX on 64-bit machine)
- + 1 + 1 = 24 */
- char fmt[64]; /* plenty big enough! */
- char *sign;
- long x;
-
- x = PyInt_AsLong(v);
- if (x == -1 && PyErr_Occurred()) {
- PyErr_Format(PyExc_TypeError, "int argument required, not %.200s",
- Py_TYPE(v)->tp_name);
- return -1;
- }
- if (x < 0 && type == 'u') {
- type = 'd';
- }
- if (x < 0 && (type == 'x' || type == 'X' || type == 'o'))
- sign = "-";
- else
- sign = "";
- if (prec < 0)
- prec = 1;
-
- if ((flags & F_ALT) &&
- (type == 'x' || type == 'X')) {
- /* When converting under %#x or %#X, there are a number
- * of issues that cause pain:
- * - when 0 is being converted, the C standard leaves off
- * the '0x' or '0X', which is inconsistent with other
- * %#x/%#X conversions and inconsistent with Python's
- * hex() function
- * - there are platforms that violate the standard and
- * convert 0 with the '0x' or '0X'
- * (Metrowerks, Compaq Tru64)
- * - there are platforms that give '0x' when converting
- * under %#X, but convert 0 in accordance with the
- * standard (OS/2 EMX)
- *
- * We can achieve the desired consistency by inserting our
- * own '0x' or '0X' prefix, and substituting %x/%X in place
- * of %#x/%#X.
- *
- * Note that this is the same approach as used in
- * formatint() in unicodeobject.c
- */
- PyOS_snprintf(fmt, sizeof(fmt), "%s0%c%%.%dl%c",
- sign, type, prec, type);
- }
- else {
- PyOS_snprintf(fmt, sizeof(fmt), "%s%%%s.%dl%c",
- sign, (flags&F_ALT) ? "#" : "",
- prec, type);
- }
-
- /* buf = '+'/'-'/'' + '0'/'0x'/'' + '[0-9]'*max(prec, len(x in octal))
- * worst case buf = '-0x' + [0-9]*prec, where prec >= 11
- */
- if (buflen <= 14 || buflen <= (size_t)3 + (size_t)prec) {
- PyErr_SetString(PyExc_OverflowError,
- "formatted integer is too long (precision too large?)");
- return -1;
- }
- if (sign[0])
- PyOS_snprintf(buf, buflen, fmt, -x);
- else
- PyOS_snprintf(buf, buflen, fmt, x);
- return (int)strlen(buf);
-}
-
-Py_LOCAL_INLINE(int)
-formatchar(char *buf, size_t buflen, PyObject *v)
-{
- /* presume that the buffer is at least 2 characters long */
- if (PyBytes_Check(v)) {
- if (!PyArg_Parse(v, "c;%c requires int or char", &buf[0]))
- return -1;
- }
- else {
- if (!PyArg_Parse(v, "b;%c requires int or char", &buf[0]))
- return -1;
- }
- buf[1] = '\0';
- return 1;
-}
-
-/* fmt%(v1,v2,...) is roughly equivalent to sprintf(fmt, v1, v2, ...)
-
- FORMATBUFLEN is the length of the buffer in which the floats, ints, &
- chars are formatted. XXX This is a magic number. Each formatting
- routine does bounds checking to ensure no overflow, but a better
- solution may be to malloc a buffer of appropriate size for each
- format. For now, the current solution is sufficient.
-*/
-#define FORMATBUFLEN (size_t)120
-
-PyObject *
-PyBytes_Format(PyObject *format, PyObject *args)
-{
- char *fmt, *res;
- Py_ssize_t arglen, argidx;
- Py_ssize_t reslen, rescnt, fmtcnt;
- int args_owned = 0;
- PyObject *result, *orig_args;
-#ifdef Py_USING_UNICODE
- PyObject *v, *w;
-#endif
- PyObject *dict = NULL;
- if (format == NULL || !PyBytes_Check(format) || args == NULL) {
- PyErr_BadInternalCall();
- return NULL;
- }
- orig_args = args;
- fmt = PyBytes_AS_STRING(format);
- fmtcnt = PyBytes_GET_SIZE(format);
- reslen = rescnt = fmtcnt + 100;
- result = PyBytes_FromStringAndSize((char *)NULL, reslen);
- if (result == NULL)
- return NULL;
- res = PyBytes_AsString(result);
- if (PyTuple_Check(args)) {
- arglen = PyTuple_GET_SIZE(args);
- argidx = 0;
- }
- else {
- arglen = -1;
- argidx = -2;
- }
- if (Py_TYPE(args)->tp_as_mapping && !PyTuple_Check(args) &&
- !PyObject_TypeCheck(args, &PyBaseString_Type))
- dict = args;
- while (--fmtcnt >= 0) {
- if (*fmt != '%') {
- if (--rescnt < 0) {
- rescnt = fmtcnt + 100;
- reslen += rescnt;
- if (_PyBytes_Resize(&result, reslen) < 0)
- return NULL;
- res = PyBytes_AS_STRING(result)
- + reslen - rescnt;
- --rescnt;
- }
- *res++ = *fmt++;
- }
- else {
- /* Got a format specifier */
- int flags = 0;
- Py_ssize_t width = -1;
- int prec = -1;
- int c = '\0';
- int fill;
- int isnumok;
- PyObject *v = NULL;
- PyObject *temp = NULL;
- char *pbuf;
- int sign;
- Py_ssize_t len;
- char formatbuf[FORMATBUFLEN];
- /* For format{float,int,char}() */
-#ifdef Py_USING_UNICODE
- char *fmt_start = fmt;
- Py_ssize_t argidx_start = argidx;
-#endif
-
- fmt++;
- if (*fmt == '(') {
- char *keystart;
- Py_ssize_t keylen;
- PyObject *key;
- int pcount = 1;
-
- if (dict == NULL) {
- PyErr_SetString(PyExc_TypeError,
- "format requires a mapping");
- goto error;
- }
- ++fmt;
- --fmtcnt;
- keystart = fmt;
- /* Skip over balanced parentheses */
- while (pcount > 0 && --fmtcnt >= 0) {
- if (*fmt == ')')
- --pcount;
- else if (*fmt == '(')
- ++pcount;
- fmt++;
- }
- keylen = fmt - keystart - 1;
- if (fmtcnt < 0 || pcount > 0) {
- PyErr_SetString(PyExc_ValueError,
- "incomplete format key");
- goto error;
- }
- key = PyBytes_FromStringAndSize(keystart,
- keylen);
- if (key == NULL)
- goto error;
- if (args_owned) {
- Py_DECREF(args);
- args_owned = 0;
- }
- args = PyObject_GetItem(dict, key);
- Py_DECREF(key);
- if (args == NULL) {
- goto error;
- }
- args_owned = 1;
- arglen = -1;
- argidx = -2;
- }
- while (--fmtcnt >= 0) {
- switch (c = *fmt++) {
- case '-': flags |= F_LJUST; continue;
- case '+': flags |= F_SIGN; continue;
- case ' ': flags |= F_BLANK; continue;
- case '#': flags |= F_ALT; continue;
- case '0': flags |= F_ZERO; continue;
- }
- break;
- }
- if (c == '*') {
- v = getnextarg(args, arglen, &argidx);
- if (v == NULL)
- goto error;
- if (!PyInt_Check(v)) {
- PyErr_SetString(PyExc_TypeError,
- "* wants int");
- goto error;
- }
- width = PyInt_AsLong(v);
- if (width < 0) {
- flags |= F_LJUST;
- width = -width;
- }
- if (--fmtcnt >= 0)
- c = *fmt++;
- }
- else if (c >= 0 && isdigit(c)) {
- width = c - '0';
- while (--fmtcnt >= 0) {
- c = Py_CHARMASK(*fmt++);
- if (!isdigit(c))
- break;
- if ((width*10) / 10 != width) {
- PyErr_SetString(
- PyExc_ValueError,
- "width too big");
- goto error;
- }
- width = width*10 + (c - '0');
- }
- }
- if (c == '.') {
- prec = 0;
- if (--fmtcnt >= 0)
- c = *fmt++;
- if (c == '*') {
- v = getnextarg(args, arglen, &argidx);
- if (v == NULL)
- goto error;
- if (!PyInt_Check(v)) {
- PyErr_SetString(
- PyExc_TypeError,
- "* wants int");
- goto error;
- }
- prec = PyInt_AsLong(v);
- if (prec < 0)
- prec = 0;
- if (--fmtcnt >= 0)
- c = *fmt++;
- }
- else if (c >= 0 && isdigit(c)) {
- prec = c - '0';
- while (--fmtcnt >= 0) {
- c = Py_CHARMASK(*fmt++);
- if (!isdigit(c))
- break;
- if ((prec*10) / 10 != prec) {
- PyErr_SetString(
- PyExc_ValueError,
- "prec too big");
- goto error;
- }
- prec = prec*10 + (c - '0');
- }
- }
- } /* prec */
- if (fmtcnt >= 0) {
- if (c == 'h' || c == 'l' || c == 'L') {
- if (--fmtcnt >= 0)
- c = *fmt++;
- }
- }
- if (fmtcnt < 0) {
- PyErr_SetString(PyExc_ValueError,
- "incomplete format");
- goto error;
- }
- if (c != '%') {
- v = getnextarg(args, arglen, &argidx);
- if (v == NULL)
- goto error;
- }
- sign = 0;
- fill = ' ';
- switch (c) {
- case '%':
- pbuf = "%";
- len = 1;
- break;
- case 's':
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(v)) {
- fmt = fmt_start;
- argidx = argidx_start;
- goto unicode;
- }
-#endif
- temp = _PyObject_Str(v);
-#ifdef Py_USING_UNICODE
- if (temp != NULL && PyUnicode_Check(temp)) {
- Py_DECREF(temp);
- fmt = fmt_start;
- argidx = argidx_start;
- goto unicode;
- }
-#endif
- /* Fall through */
- case 'r':
- if (c == 'r')
- temp = PyObject_Repr(v);
- if (temp == NULL)
- goto error;
- if (!PyBytes_Check(temp)) {
- PyErr_SetString(PyExc_TypeError,
- "%s argument has non-string str()");
- Py_DECREF(temp);
- goto error;
- }
- pbuf = PyBytes_AS_STRING(temp);
- len = PyBytes_GET_SIZE(temp);
- if (prec >= 0 && len > prec)
- len = prec;
- break;
- case 'i':
- case 'd':
- case 'u':
- case 'o':
- case 'x':
- case 'X':
- if (c == 'i')
- c = 'd';
- isnumok = 0;
- if (PyNumber_Check(v)) {
- PyObject *iobj=NULL;
-
- if (PyInt_Check(v) || (PyLong_Check(v))) {
- iobj = v;
- Py_INCREF(iobj);
- }
- else {
- iobj = PyNumber_Int(v);
- if (iobj==NULL) iobj = PyNumber_Long(v);
- }
- if (iobj!=NULL) {
- if (PyInt_Check(iobj)) {
- isnumok = 1;
- pbuf = formatbuf;
- len = formatint(pbuf,
- sizeof(formatbuf),
- flags, prec, c, iobj);
- Py_DECREF(iobj);
- if (len < 0)
- goto error;
- sign = 1;
- }
- else if (PyLong_Check(iobj)) {
- int ilen;
-
- isnumok = 1;
- temp = _PyBytes_FormatLong(iobj, flags,
- prec, c, &pbuf, &ilen);
- Py_DECREF(iobj);
- len = ilen;
- if (!temp)
- goto error;
- sign = 1;
- }
- else {
- Py_DECREF(iobj);
- }
- }
- }
- if (!isnumok) {
- PyErr_Format(PyExc_TypeError,
- "%%%c format: a number is required, "
- "not %.200s", c, Py_TYPE(v)->tp_name);
- goto error;
- }
- if (flags & F_ZERO)
- fill = '0';
- break;
- case 'e':
- case 'E':
- case 'f':
- case 'F':
- case 'g':
- case 'G':
- if (c == 'F')
- c = 'f';
- pbuf = formatbuf;
- len = formatfloat(pbuf, sizeof(formatbuf),
- flags, prec, c, v);
- if (len < 0)
- goto error;
- sign = 1;
- if (flags & F_ZERO)
- fill = '0';
- break;
- case 'c':
-#ifdef Py_USING_UNICODE
- if (PyUnicode_Check(v)) {
- fmt = fmt_start;
- argidx = argidx_start;
- goto unicode;
- }
-#endif
- pbuf = formatbuf;
- len = formatchar(pbuf, sizeof(formatbuf), v);
- if (len < 0)
- goto error;
- break;
- default:
- PyErr_Format(PyExc_ValueError,
- "unsupported format character '%c' (0x%x) "
- "at index %zd",
- c, c,
- (Py_ssize_t)(fmt - 1 -
- PyBytes_AsString(format)));
- goto error;
- }
- if (sign) {
- if (*pbuf == '-' || *pbuf == '+') {
- sign = *pbuf++;
- len--;
- }
- else if (flags & F_SIGN)
- sign = '+';
- else if (flags & F_BLANK)
- sign = ' ';
- else
- sign = 0;
- }
- if (width < len)
- width = len;
- if (rescnt - (sign != 0) < width) {
- reslen -= rescnt;
- rescnt = width + fmtcnt + 100;
- reslen += rescnt;
- if (reslen < 0) {
- Py_DECREF(result);
- Py_XDECREF(temp);
- return PyErr_NoMemory();
- }
- if (_PyBytes_Resize(&result, reslen) < 0) {
- Py_XDECREF(temp);
- return NULL;
- }
- res = PyBytes_AS_STRING(result)
- + reslen - rescnt;
- }
- if (sign) {
- if (fill != ' ')
- *res++ = sign;
- rescnt--;
- if (width > len)
- width--;
- }
- if ((flags & F_ALT) && (c == 'x' || c == 'X')) {
- assert(pbuf[0] == '0');
- assert(pbuf[1] == c);
- if (fill != ' ') {
- *res++ = *pbuf++;
- *res++ = *pbuf++;
- }
- rescnt -= 2;
- width -= 2;
- if (width < 0)
- width = 0;
- len -= 2;
- }
- if (width > len && !(flags & F_LJUST)) {
- do {
- --rescnt;
- *res++ = fill;
- } while (--width > len);
- }
- if (fill == ' ') {
- if (sign)
- *res++ = sign;
- if ((flags & F_ALT) &&
- (c == 'x' || c == 'X')) {
- assert(pbuf[0] == '0');
- assert(pbuf[1] == c);
- *res++ = *pbuf++;
- *res++ = *pbuf++;
- }
- }
- Py_MEMCPY(res, pbuf, len);
- res += len;
- rescnt -= len;
- while (--width >= len) {
- --rescnt;
- *res++ = ' ';
- }
- if (dict && (argidx < arglen) && c != '%') {
- PyErr_SetString(PyExc_TypeError,
- "not all arguments converted during string formatting");
- Py_XDECREF(temp);
- goto error;
- }
- Py_XDECREF(temp);
- } /* '%' */
- } /* until end */
- if (argidx < arglen && !dict) {
- PyErr_SetString(PyExc_TypeError,
- "not all arguments converted during string formatting");
- goto error;
- }
- if (args_owned) {
- Py_DECREF(args);
- }
- _PyBytes_Resize(&result, reslen - rescnt);
- return result;
-
-#ifdef Py_USING_UNICODE
- unicode:
- if (args_owned) {
- Py_DECREF(args);
- args_owned = 0;
- }
- /* Fiddle args right (remove the first argidx arguments) */
- if (PyTuple_Check(orig_args) && argidx > 0) {
- PyObject *v;
- Py_ssize_t n = PyTuple_GET_SIZE(orig_args) - argidx;
- v = PyTuple_New(n);
- if (v == NULL)
- goto error;
- while (--n >= 0) {
- PyObject *w = PyTuple_GET_ITEM(orig_args, n + argidx);
- Py_INCREF(w);
- PyTuple_SET_ITEM(v, n, w);
- }
- args = v;
- } else {
- Py_INCREF(orig_args);
- args = orig_args;
- }
- args_owned = 1;
- /* Take what we have of the result and let the Unicode formatting
- function format the rest of the input. */
- rescnt = res - PyBytes_AS_STRING(result);
- if (_PyBytes_Resize(&result, rescnt))
- goto error;
- fmtcnt = PyBytes_GET_SIZE(format) - \
- (fmt - PyBytes_AS_STRING(format));
- format = PyUnicode_Decode(fmt, fmtcnt, NULL, NULL);
- if (format == NULL)
- goto error;
- v = PyUnicode_Format(format, args);
- Py_DECREF(format);
- if (v == NULL)
- goto error;
- /* Paste what we have (result) to what the Unicode formatting
- function returned (v) and return the result (or error) */
- w = PyUnicode_Concat(result, v);
- Py_DECREF(result);
- Py_DECREF(v);
- Py_DECREF(args);
- return w;
-#endif /* Py_USING_UNICODE */
-
- error:
- Py_DECREF(result);
- if (args_owned) {
- Py_DECREF(args);
- }
- return NULL;
-}
-
-void
-PyString_InternInPlace(PyObject **p)
-{
- register PyBytesObject *s = (PyBytesObject *)(*p);
- PyObject *t;
- if (s == NULL || !PyBytes_Check(s))
- Py_FatalError("PyString_InternInPlace: strings only please!");
- /* If it's a string subclass, we don't really know what putting
- it in the interned dict might do. */
- if (!PyBytes_CheckExact(s))
- return;
- if (PyBytes_CHECK_INTERNED(s))
- return;
- if (interned == NULL) {
- interned = PyDict_New();
- if (interned == NULL) {
- PyErr_Clear(); /* Don't leave an exception */
- return;
- }
- }
- t = PyDict_GetItem(interned, (PyObject *)s);
- if (t) {
- Py_INCREF(t);
- Py_DECREF(*p);
- *p = t;
- return;
- }
-
- if (PyDict_SetItem(interned, (PyObject *)s, (PyObject *)s) < 0) {
- PyErr_Clear();
- return;
- }
- /* The two references in interned are not counted by refcnt.
- The string deallocator will take care of this */
- Py_REFCNT(s) -= 2;
- PyBytes_CHECK_INTERNED(s) = SSTATE_INTERNED_MORTAL;
-}
-
-void
-PyString_InternImmortal(PyObject **p)
-{
- PyString_InternInPlace(p);
- if (PyBytes_CHECK_INTERNED(*p) != SSTATE_INTERNED_IMMORTAL) {
- PyBytes_CHECK_INTERNED(*p) = SSTATE_INTERNED_IMMORTAL;
- Py_INCREF(*p);
- }
-}
-
-
-PyObject *
-PyString_InternFromString(const char *cp)
-{
- PyObject *s = PyBytes_FromString(cp);
- if (s == NULL)
- return NULL;
- PyString_InternInPlace(&s);
- return s;
-}
-
-void
-PyString_Fini(void)
-{
- int i;
- for (i = 0; i < UCHAR_MAX + 1; i++) {
- Py_XDECREF(characters[i]);
- characters[i] = NULL;
- }
- Py_XDECREF(nullstring);
- nullstring = NULL;
-}
-
-void _Py_ReleaseInternedStrings(void)
-{
- PyObject *keys;
- PyBytesObject *s;
- Py_ssize_t i, n;
- Py_ssize_t immortal_size = 0, mortal_size = 0;
-
- if (interned == NULL || !PyDict_Check(interned))
- return;
- keys = PyDict_Keys(interned);
- if (keys == NULL || !PyList_Check(keys)) {
- PyErr_Clear();
- return;
- }
-
- /* Since _Py_ReleaseInternedStrings() is intended to help a leak
- detector, interned strings are not forcibly deallocated; rather, we
- give them their stolen references back, and then clear and DECREF
- the interned dict. */
-
- n = PyList_GET_SIZE(keys);
- fprintf(stderr, "releasing %" PY_FORMAT_SIZE_T "d interned strings\n",
- n);
- for (i = 0; i < n; i++) {
- s = (PyBytesObject *) PyList_GET_ITEM(keys, i);
- switch (s->ob_sstate) {
- case SSTATE_NOT_INTERNED:
- /* XXX Shouldn't happen */
- break;
- case SSTATE_INTERNED_IMMORTAL:
- Py_REFCNT(s) += 1;
- immortal_size += Py_SIZE(s);
- break;
- case SSTATE_INTERNED_MORTAL:
- Py_REFCNT(s) += 2;
- mortal_size += Py_SIZE(s);
- break;
- default:
- Py_FatalError("Inconsistent interned string state.");
- }
- s->ob_sstate = SSTATE_NOT_INTERNED;
- }
- fprintf(stderr, "total size of all interned strings: "
- "%" PY_FORMAT_SIZE_T "d/%" PY_FORMAT_SIZE_T "d "
- "mortal/immortal\n", mortal_size, immortal_size);
- Py_DECREF(keys);
- PyDict_Clear(interned);
- Py_DECREF(interned);
- interned = NULL;
-}
Modified: python/branches/tlee-ast-optimize/Objects/fileobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/fileobject.c (original)
+++ python/branches/tlee-ast-optimize/Objects/fileobject.c Fri Jun 13 16:43:12 2008
@@ -453,7 +453,7 @@
PyObject *str, *oerrors;
assert(PyFile_Check(f));
- str = PyBytes_FromString(enc);
+ str = PyString_FromString(enc);
if (!str)
return 0;
if (errors) {
@@ -2321,9 +2321,9 @@
#ifdef Py_USING_UNICODE
if ((flags & Py_PRINT_RAW) &&
PyUnicode_Check(v) && enc != Py_None) {
- char *cenc = PyBytes_AS_STRING(enc);
+ char *cenc = PyString_AS_STRING(enc);
char *errors = fobj->f_errors == Py_None ?
- "strict" : PyBytes_AS_STRING(fobj->f_errors);
+ "strict" : PyString_AS_STRING(fobj->f_errors);
value = PyUnicode_AsEncodedString(v, cenc, errors);
if (value == NULL)
return -1;
Modified: python/branches/tlee-ast-optimize/Objects/listobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/listobject.c (original)
+++ python/branches/tlee-ast-optimize/Objects/listobject.c Fri Jun 13 16:43:12 2008
@@ -45,7 +45,16 @@
* system realloc().
* The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...
*/
- new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6) + newsize;
+ new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6);
+
+ /* check for integer overflow */
+ if (new_allocated > PY_SIZE_MAX - newsize) {
+ PyErr_NoMemory();
+ return -1;
+ } else {
+ new_allocated += newsize;
+ }
+
if (newsize == 0)
new_allocated = 0;
items = self->ob_item;
@@ -118,8 +127,9 @@
return NULL;
}
nbytes = size * sizeof(PyObject *);
- /* Check for overflow */
- if (nbytes / sizeof(PyObject *) != (size_t)size)
+ /* Check for overflow without an actual overflow,
+ * which can cause compiler to optimise out */
+ if (size > PY_SIZE_MAX / sizeof(PyObject *))
return PyErr_NoMemory();
if (numfree) {
numfree--;
@@ -1407,6 +1417,10 @@
* we don't care what's in the block.
*/
merge_freemem(ms);
+ if (need > PY_SSIZE_T_MAX / sizeof(PyObject*)) {
+ PyErr_NoMemory();
+ return -1;
+ }
ms->a = (PyObject **)PyMem_Malloc(need * sizeof(PyObject*));
if (ms->a) {
ms->alloced = need;
@@ -2589,6 +2603,8 @@
step = -step;
}
+ assert(slicelength <= PY_SIZE_MAX / sizeof(PyObject*));
+
garbage = (PyObject**)
PyMem_MALLOC(slicelength*sizeof(PyObject*));
if (!garbage) {
Modified: python/branches/tlee-ast-optimize/Objects/obmalloc.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/obmalloc.c (original)
+++ python/branches/tlee-ast-optimize/Objects/obmalloc.c Fri Jun 13 16:43:12 2008
@@ -526,9 +526,9 @@
numarenas = maxarenas ? maxarenas << 1 : INITIAL_ARENA_OBJECTS;
if (numarenas <= maxarenas)
return NULL; /* overflow */
- nbytes = numarenas * sizeof(*arenas);
- if (nbytes / sizeof(*arenas) != numarenas)
+ if (numarenas > PY_SIZE_MAX / sizeof(*arenas))
return NULL; /* overflow */
+ nbytes = numarenas * sizeof(*arenas);
arenaobj = (struct arena_object *)realloc(arenas, nbytes);
if (arenaobj == NULL)
return NULL;
Modified: python/branches/tlee-ast-optimize/Objects/setobject.c
==============================================================================
--- python/branches/tlee-ast-optimize/Objects/setobject.c (original)
+++ python/branches/tlee-ast-optimize/Objects/setobject.c Fri Jun 13 16:43:12 2008
@@ -1312,6 +1312,9 @@
Py_ssize_t i;
PyObject *result = (PyObject *)so;
+ if (PyTuple_GET_SIZE(args) == 0)
+ return set_copy(so);
+
Py_INCREF(so);
for (i=0 ; i<PyTuple_GET_SIZE(args) ; i++) {
PyObject *other = PyTuple_GET_ITEM(args, i);
@@ -1492,11 +1495,16 @@
}
static PyObject *
-set_difference_update(PySetObject *so, PyObject *other)
+set_difference_update(PySetObject *so, PyObject *args)
{
- if (set_difference_update_internal(so, other) != -1)
- Py_RETURN_NONE;
- return NULL;
+ Py_ssize_t i;
+
+ for (i=0 ; i<PyTuple_GET_SIZE(args) ; i++) {
+ PyObject *other = PyTuple_GET_ITEM(args, i);
+ if (set_difference_update_internal(so, other) == -1)
+ return NULL;
+ }
+ Py_RETURN_NONE;
}
PyDoc_STRVAR(difference_update_doc,
@@ -1554,10 +1562,34 @@
return result;
}
+static PyObject *
+set_difference_multi(PySetObject *so, PyObject *args)
+{
+ Py_ssize_t i;
+ PyObject *result, *other;
+
+ if (PyTuple_GET_SIZE(args) == 0)
+ return set_copy(so);
+
+ other = PyTuple_GET_ITEM(args, 0);
+ result = set_difference(so, other);
+ if (result == NULL)
+ return NULL;
+
+ for (i=1 ; i<PyTuple_GET_SIZE(args) ; i++) {
+ other = PyTuple_GET_ITEM(args, i);
+ if (set_difference_update_internal((PySetObject *)result, other) == -1) {
+ Py_DECREF(result);
+ return NULL;
+ }
+ }
+ return result;
+}
+
PyDoc_STRVAR(difference_doc,
-"Return the difference of two sets as a new set.\n\
+"Return the difference of two or more sets as a new set.\n\
\n\
-(i.e. all elements that are in this set but not the other.)");
+(i.e. all elements that are in this set but not the others.)");
static PyObject *
set_sub(PySetObject *so, PyObject *other)
{
@@ -1571,16 +1603,12 @@
static PyObject *
set_isub(PySetObject *so, PyObject *other)
{
- PyObject *result;
-
if (!PyAnySet_Check(other)) {
Py_INCREF(Py_NotImplemented);
return Py_NotImplemented;
}
- result = set_difference_update(so, other);
- if (result == NULL)
+ if (set_difference_update_internal(so, other) == -1)
return NULL;
- Py_DECREF(result);
Py_INCREF(so);
return (PyObject *)so;
}
@@ -1975,9 +2003,9 @@
copy_doc},
{"discard", (PyCFunction)set_discard, METH_O,
discard_doc},
- {"difference", (PyCFunction)set_difference, METH_O,
+ {"difference", (PyCFunction)set_difference_multi, METH_VARARGS,
difference_doc},
- {"difference_update", (PyCFunction)set_difference_update, METH_O,
+ {"difference_update", (PyCFunction)set_difference_update, METH_VARARGS,
difference_update_doc},
{"intersection",(PyCFunction)set_intersection_multi, METH_VARARGS,
intersection_doc},
@@ -2104,7 +2132,7 @@
contains_doc},
{"copy", (PyCFunction)frozenset_copy, METH_NOARGS,
copy_doc},
- {"difference", (PyCFunction)set_difference, METH_O,
+ {"difference", (PyCFunction)set_difference_multi, METH_VARARGS,
difference_doc},
{"intersection",(PyCFunction)set_intersection_multi, METH_VARARGS,
intersection_doc},
Modified: python/branches/tlee-ast-optimize/PC/VC6/pythoncore.dsp
==============================================================================
--- python/branches/tlee-ast-optimize/PC/VC6/pythoncore.dsp (original)
+++ python/branches/tlee-ast-optimize/PC/VC6/pythoncore.dsp Fri Jun 13 16:43:12 2008
@@ -241,7 +241,7 @@
# End Source File
# Begin Source File
-SOURCE=..\..\Objects\bytesobject.c
+SOURCE=..\..\Objects\stringobject.c
# End Source File
# Begin Source File
Modified: python/branches/tlee-ast-optimize/PC/VS7.1/pythoncore.vcproj
==============================================================================
--- python/branches/tlee-ast-optimize/PC/VS7.1/pythoncore.vcproj (original)
+++ python/branches/tlee-ast-optimize/PC/VS7.1/pythoncore.vcproj Fri Jun 13 16:43:12 2008
@@ -446,7 +446,7 @@
RelativePath="..\..\Objects\bytearrayobject.c">
</File>
<File
- RelativePath="..\..\Objects\bytesobject.c">
+ RelativePath="..\..\Objects\stringobject.c">
</File>
<File
RelativePath="..\..\Objects\bytes_methods.c">
Modified: python/branches/tlee-ast-optimize/PC/VS8.0/pythoncore.vcproj
==============================================================================
--- python/branches/tlee-ast-optimize/PC/VS8.0/pythoncore.vcproj (original)
+++ python/branches/tlee-ast-optimize/PC/VS8.0/pythoncore.vcproj Fri Jun 13 16:43:12 2008
@@ -1379,7 +1379,7 @@
>
</File>
<File
- RelativePath="..\..\Objects\bytesobject.c"
+ RelativePath="..\..\Objects\stringobject.c"
>
</File>
<File
Modified: python/branches/tlee-ast-optimize/PCbuild/pythoncore.vcproj
==============================================================================
--- python/branches/tlee-ast-optimize/PCbuild/pythoncore.vcproj (original)
+++ python/branches/tlee-ast-optimize/PCbuild/pythoncore.vcproj Fri Jun 13 16:43:12 2008
@@ -1379,7 +1379,7 @@
>
</File>
<File
- RelativePath="..\Objects\bytesobject.c"
+ RelativePath="..\Objects\stringobject.c"
>
</File>
<File
Modified: python/branches/tlee-ast-optimize/Parser/node.c
==============================================================================
--- python/branches/tlee-ast-optimize/Parser/node.c (original)
+++ python/branches/tlee-ast-optimize/Parser/node.c Fri Jun 13 16:43:12 2008
@@ -91,6 +91,9 @@
if (current_capacity < 0 || required_capacity < 0)
return E_OVERFLOW;
if (current_capacity < required_capacity) {
+ if (required_capacity > PY_SIZE_MAX / sizeof(node)) {
+ return E_NOMEM;
+ }
n = n1->n_child;
n = (node *) PyObject_REALLOC(n,
required_capacity * sizeof(node));
Modified: python/branches/tlee-ast-optimize/Python/asdl.c
==============================================================================
--- python/branches/tlee-ast-optimize/Python/asdl.c (original)
+++ python/branches/tlee-ast-optimize/Python/asdl.c Fri Jun 13 16:43:12 2008
@@ -5,8 +5,22 @@
asdl_seq_new(int size, PyArena *arena)
{
asdl_seq *seq = NULL;
- size_t n = sizeof(asdl_seq) +
- (size ? (sizeof(void *) * (size - 1)) : 0);
+ size_t n = (size ? (sizeof(void *) * (size - 1)) : 0);
+
+ /* check size is sane */
+ if (size < 0 || size == INT_MIN ||
+ (size && ((size - 1) > (PY_SIZE_MAX / sizeof(void *))))) {
+ PyErr_NoMemory();
+ return NULL;
+ }
+
+ /* check if size can be added safely */
+ if (n > PY_SIZE_MAX - sizeof(asdl_seq)) {
+ PyErr_NoMemory();
+ return NULL;
+ }
+
+ n += sizeof(asdl_seq);
seq = (asdl_seq *)PyArena_Malloc(arena, n);
if (!seq) {
@@ -22,8 +36,22 @@
asdl_int_seq_new(int size, PyArena *arena)
{
asdl_int_seq *seq = NULL;
- size_t n = sizeof(asdl_seq) +
- (size ? (sizeof(int) * (size - 1)) : 0);
+ size_t n = (size ? (sizeof(void *) * (size - 1)) : 0);
+
+ /* check size is sane */
+ if (size < 0 || size == INT_MIN ||
+ (size && ((size - 1) > (PY_SIZE_MAX / sizeof(void *))))) {
+ PyErr_NoMemory();
+ return NULL;
+ }
+
+ /* check if size can be added safely */
+ if (n > PY_SIZE_MAX - sizeof(asdl_seq)) {
+ PyErr_NoMemory();
+ return NULL;
+ }
+
+ n += sizeof(asdl_seq);
seq = (asdl_int_seq *)PyArena_Malloc(arena, n);
if (!seq) {
Modified: python/branches/tlee-ast-optimize/Python/ast.c
==============================================================================
--- python/branches/tlee-ast-optimize/Python/ast.c (original)
+++ python/branches/tlee-ast-optimize/Python/ast.c Fri Jun 13 16:43:12 2008
@@ -3200,6 +3200,9 @@
buf = (char *)s;
u = NULL;
} else {
+ /* check for integer overflow */
+ if (len > PY_SIZE_MAX / 4)
+ return NULL;
/* "\XX" may become "\u005c\uHHLL" (12 bytes) */
u = PyString_FromStringAndSize((char *)NULL, len * 4);
if (u == NULL)
Modified: python/branches/tlee-ast-optimize/Python/bltinmodule.c
==============================================================================
--- python/branches/tlee-ast-optimize/Python/bltinmodule.c (original)
+++ python/branches/tlee-ast-optimize/Python/bltinmodule.c Fri Jun 13 16:43:12 2008
@@ -2835,11 +2835,43 @@
PyString_AS_STRING(item)[0];
} else {
/* do we need more space? */
- Py_ssize_t need = j + reslen + len-i-1;
+ Py_ssize_t need = j;
+
+ /* calculate space requirements while checking for overflow */
+ if (need > PY_SSIZE_T_MAX - reslen) {
+ Py_DECREF(item);
+ goto Fail_1;
+ }
+
+ need += reslen;
+
+ if (need > PY_SSIZE_T_MAX - len) {
+ Py_DECREF(item);
+ goto Fail_1;
+ }
+
+ need += len;
+
+ if (need <= i) {
+ Py_DECREF(item);
+ goto Fail_1;
+ }
+
+ need = need - i - 1;
+
+ assert(need >= 0);
+ assert(outlen >= 0);
+
if (need > outlen) {
/* overallocate, to avoid reallocations */
- if (need<2*outlen)
+ if (outlen > PY_SSIZE_T_MAX / 2) {
+ Py_DECREF(item);
+ return NULL;
+ }
+
+ if (need<2*outlen) {
need = 2*outlen;
+ }
if (_PyString_Resize(&result, need)) {
Py_DECREF(item);
return NULL;
@@ -2931,11 +2963,31 @@
else {
/* do we need more space? */
Py_ssize_t need = j + reslen + len - i - 1;
+
+ /* check that didnt overflow */
+ if ((j > PY_SSIZE_T_MAX - reslen) ||
+ ((j + reslen) > PY_SSIZE_T_MAX - len) ||
+ ((j + reslen + len) < i) ||
+ ((j + reslen + len - i) <= 0)) {
+ Py_DECREF(item);
+ return NULL;
+ }
+
+ assert(need >= 0);
+ assert(outlen >= 0);
+
if (need > outlen) {
/* overallocate,
to avoid reallocations */
- if (need < 2 * outlen)
- need = 2 * outlen;
+ if (need < 2 * outlen) {
+ if (outlen > PY_SSIZE_T_MAX / 2) {
+ Py_DECREF(item);
+ return NULL;
+ } else {
+ need = 2 * outlen;
+ }
+ }
+
if (PyUnicode_Resize(
&result, need) < 0) {
Py_DECREF(item);
Modified: python/branches/tlee-ast-optimize/Python/compile.c
==============================================================================
--- python/branches/tlee-ast-optimize/Python/compile.c (original)
+++ python/branches/tlee-ast-optimize/Python/compile.c Fri Jun 13 16:43:12 2008
@@ -217,6 +217,10 @@
return ident; /* Don't mangle if class is just underscores */
}
plen = strlen(p);
+
+ assert(1 <= PY_SSIZE_T_MAX - nlen);
+ assert(1 + nlen <= PY_SSIZE_T_MAX - plen);
+
ident = PyString_FromStringAndSize(NULL, 1 + nlen + plen);
if (!ident)
return 0;
@@ -667,6 +671,12 @@
size_t oldsize, newsize;
oldsize = b->b_ialloc * sizeof(struct instr);
newsize = oldsize << 1;
+
+ if (oldsize > (PY_SIZE_MAX >> 1)) {
+ PyErr_NoMemory();
+ return -1;
+ }
+
if (newsize == 0) {
PyErr_NoMemory();
return -1;
@@ -3527,6 +3537,10 @@
a->a_lnotab = PyString_FromStringAndSize(NULL, DEFAULT_LNOTAB_SIZE);
if (!a->a_lnotab)
return 0;
+ if (nblocks > PY_SIZE_MAX / sizeof(basicblock *)) {
+ PyErr_NoMemory();
+ return 0;
+ }
a->a_postorder = (basicblock **)PyObject_Malloc(
sizeof(basicblock *) * nblocks);
if (!a->a_postorder) {
@@ -3638,10 +3652,14 @@
nbytes = a->a_lnotab_off + 2 * ncodes;
len = PyString_GET_SIZE(a->a_lnotab);
if (nbytes >= len) {
- if (len * 2 < nbytes)
+ if ((len <= INT_MAX / 2) && (len * 2 < nbytes))
len = nbytes;
- else
+ else if (len <= INT_MAX / 2)
len *= 2;
+ else {
+ PyErr_NoMemory();
+ return 0;
+ }
if (_PyString_Resize(&a->a_lnotab, len) < 0)
return 0;
}
@@ -3660,10 +3678,14 @@
nbytes = a->a_lnotab_off + 2 * ncodes;
len = PyString_GET_SIZE(a->a_lnotab);
if (nbytes >= len) {
- if (len * 2 < nbytes)
+ if ((len <= INT_MAX / 2) && len * 2 < nbytes)
len = nbytes;
- else
+ else if (len <= INT_MAX / 2)
len *= 2;
+ else {
+ PyErr_NoMemory();
+ return 0;
+ }
if (_PyString_Resize(&a->a_lnotab, len) < 0)
return 0;
}
@@ -3722,6 +3744,8 @@
if (i->i_lineno && !assemble_lnotab(a, i))
return 0;
if (a->a_offset + size >= len) {
+ if (len > PY_SSIZE_T_MAX / 2)
+ return 0;
if (_PyString_Resize(&a->a_bytecode, len * 2) < 0)
return 0;
}
Modified: python/branches/tlee-ast-optimize/setup.py
==============================================================================
--- python/branches/tlee-ast-optimize/setup.py (original)
+++ python/branches/tlee-ast-optimize/setup.py Fri Jun 13 16:43:12 2008
@@ -1236,6 +1236,58 @@
# Thomas Heller's _ctypes module
self.detect_ctypes(inc_dirs, lib_dirs)
+ # Richard Oudkerk's multiprocessing module
+ if platform == 'win32': # Windows
+ macros = dict()
+ libraries = ['ws2_32']
+
+ elif platform == 'darwin': # Mac OSX
+ macros = dict(
+ HAVE_SEM_OPEN=1,
+ HAVE_SEM_TIMEDWAIT=0,
+ HAVE_FD_TRANSFER=1,
+ HAVE_BROKEN_SEM_GETVALUE=1
+ )
+ libraries = []
+
+ elif platform == 'cygwin': # Cygwin
+ macros = dict(
+ HAVE_SEM_OPEN=1,
+ HAVE_SEM_TIMEDWAIT=1,
+ HAVE_FD_TRANSFER=0,
+ HAVE_BROKEN_SEM_UNLINK=1
+ )
+ libraries = []
+ else: # Linux and other unices
+ macros = dict(
+ HAVE_SEM_OPEN=1,
+ HAVE_SEM_TIMEDWAIT=1,
+ HAVE_FD_TRANSFER=1
+ )
+ libraries = ['rt']
+
+ if platform == 'win32':
+ multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+ '_multiprocessing/semaphore.c',
+ '_multiprocessing/pipe_connection.c',
+ '_multiprocessing/socket_connection.c',
+ '_multiprocessing/win32_functions.c'
+ ]
+
+ else:
+ multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+ '_multiprocessing/socket_connection.c'
+ ]
+
+ if macros.get('HAVE_SEM_OPEN', False):
+ multiprocessing_srcs.append('_multiprocessing/semaphore.c')
+
+ exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+ define_macros=macros.items(),
+ include_dirs=["Modules/_multiprocessing"]))
+ # End multiprocessing
+
+
# Platform-specific libraries
if platform == 'linux2':
# Linux-specific modules
More information about the Python-checkins
mailing list