capi-sig September 2018

capi-sig@python.org

19 participants
24 discussions

ANN: expy 0.6.5 released!
by Yingjie Lan 16 Apr '22

16 Apr '22

EXPY is an express way to extend Python! EXPY provides a way to extend python in an elegant way. For more information and a tutorial, see: http://expy.sourceforge.net/ What's new: 1. Correct treatment of __init__ method. 2. Give warnings of missing Py_INCREF on appropriate special type methods. 3. Documentation update. Cheers, Yingjie

3 3

ANNOUNCE: PyBindGen 0.8
by Gustavo Carneiro 10 Mar '22

10 Mar '22

The other day I released PyBindGen 0.8. Main news is that it features a new experimental header file scanner based on pygccxml (i.e., it's similar to py++ in scope, if not in maturity, but does not use boost.pythonunderneath). == What == PyBindGen is a Python module that is geared to generating C/C++ code that binds a C/C++ library for Python. It does so without extensive use of either C++ templates or C pre-processor macros. It has modular handling of C/C++ types, and can be easily extended with Python plugins. The generated code is almost as clean as what a human programmer would write, and does not depend on any library or header files besides Python itself. == Where == https://launchpad.net/pybindgen/ == NEWS == - Support C++ instance attributes through getter/setter methods - Support functions as methods of C++ classes - Support the PyObject* type - Support unsigned int, C strings (char*) (from Mark Lee) - Add basic support for enum types - New experimental automatic module generator based on C/C++ header file scanner and annotations in comments, using pygccxml - Some bug fixes -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert

2 1

C API equivalent to abc.ABCMeta.register?
by Daniel Stutzbach 02 Mar '22

02 Mar '22

I'm the author of an extension module (blist) that provides a type that fits the MutableSequence API. Is there a canonical way for me to register the type as a MutableSequence from the C API? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>

3 2

Software Development - Application Development - Game Development
by Gary 10 Jan '22

10 Jan '22

You are receiving this email because we wish you to use our cost-effective IT services. We are a China based Custom Software Application Development provider. We offer full cycle custom software programming services, from product idea, software development to support and enhancement. We employ a large pool of software engineers coming from different backgrounds. We are able to balance product development efforts and project duration to your business needs. Core Services: Custom Application Development Custom Software Development iPhone Application Development iPad Application Development Mobile Application Development Android Mobile Application Development Soft Product Development Game Design & Development Game Testing & Quality Assurance Cloud Computing Solutions: CRM Software Application Development ERP Software Application Development Enterprise Portal Solution Sales Force Automation Solution Game Design & Development Knowledge Management Solution Workflow Management Solution SharePoint Development Services Microsoft Online Services Microsoft Azure Services Best regards, Gary ITVIATSA Software Development Contact: ibsoftware(a)yeah.net Pls send address to larryremove123(a)msn.com for remove

2 2

[capi-sig]fixedint patch, proof of concept working now
by Neil Schemenauer 19 Sep '18

19 Sep '18

Hi, It has taken a fair amount of work but I have mostly gotten tagged pointers working for small ints (currently 63 bits on 64-bit platform). The configure option is --with-fixedint. I'm trying to release early and often. The whole test suite runs without crashing, which feels like some kind of milestone to me. The following tests still fail: test_ctypes test_fcntl test_fileio test_gdb test_inspect test_io test_repl test_socket test_sqlite test_unicode test_userstring Latest code in here: https://github.com/nascheme/cpython/tree/tagged_int Unfortunately, there is a net slowdown with the fixedint option enabled. Full PGO benchmarks from pyperformance are below. I'm hoping there is still some good work to be done reducing the number of times fixed ints need to heap allocated. I suspect that is why the pickle_list and pickle_dict are slower. I should also try to measure the memory usage difference. The fixedint version should use less RAM. Here is a Linux perf report for the pickle_list benchmark: http://python.ca/nas/python/perf-fixedint-pickle-list.txt The addition of the extra test + jmp instructions to INCREF/DECREF are hurting a fair bit. I'm not sure there is anything to be done there. Based on the Linux perf results, I have the suspicion that the extra instructions for INCREF/DECREF/_Py_TYPE are blowing up the size of _PyEval_EvalFrameDefault. I need to investigate that more. BTW, Linux perf is amazing. Anyone who does low-level optimization work should study it. I did consider trying to use a second tag for short strings. Not sure it will help too much as some quick analysis shows that only 25% of strings used by PyDict_GetItem are short enough to fit. This morning I dreamed up new idea: analyze normal Python programs and build a list of strings commonly used for PyDict_GetItem. They will be strings like "self", builtin functions names, etc. Then use a tagged pointer to hold these common strings. I.e. a tag to denote a string (or interned symbol, in Lisp speak) and an integer which is the offset into the fixed array of interned strings. The savings would have to come from avoiding the INCREF/DECREF accounting of refcounts on those strings. Instead of fixed set of strings, perhaps we could make the intern process dynamically allocate the tag IDs. We could have a specialized lookdict that works for dicts containing only interned strings. $ ./python -m perf compare_to -G ../cpython-profile-tagged-off/base4.json fixedint5.json --min-spee d 5 Slower (24): - pickle_list: 3.06 us +- 0.04 us -> 3.74 us +- 0.03 us: 1.22x slower (+22%) - pickle_dict: 22.2 us +- 0.1 us -> 26.2 us +- 0.2 us: 1.18x slower (+18%) - raytrace: 501 ms +- 5 ms -> 565 ms +- 6 ms: 1.13x slower (+13%) - crypto_pyaes: 113 ms +- 1 ms -> 126 ms +- 0 ms: 1.12x slower (+12%) - logging_silent: 210 ns +- 4 ns -> 234 ns +- 3 ns: 1.11x slower (+11%) - telco: 6.00 ms +- 0.09 ms -> 6.68 ms +- 0.14 ms: 1.11x slower (+11%) - float: 111 ms +- 2 ms -> 123 ms +- 1 ms: 1.11x slower (+11%) - nbody: 122 ms +- 1 ms -> 135 ms +- 2 ms: 1.10x slower (+10%) - mako: 17.1 ms +- 0.1 ms -> 18.8 ms +- 0.1 ms: 1.10x slower (+10%) - json_dumps: 12.3 ms +- 0.2 ms -> 13.5 ms +- 0.1 ms: 1.10x slower (+10%) - scimark_monte_carlo: 103 ms +- 2 ms -> 113 ms +- 1 ms: 1.10x slower (+10%) - pickle_pure_python: 467 us +- 3 us -> 508 us +- 6 us: 1.09x slower (+9%) - logging_format: 10.2 us +- 0.1 us -> 11.1 us +- 2.2 us: 1.09x slower (+9%) - chameleon: 9.27 ms +- 0.09 ms -> 10.1 ms +- 0.1 ms: 1.09x slower (+9%) - sqlalchemy_imperative: 30.4 ms +- 0.8 ms -> 32.9 ms +- 0.9 ms: 1.08x slower (+8%) - django_template: 122 ms +- 2 ms -> 131 ms +- 2 ms: 1.08x slower (+8%) - sympy_str: 184 ms +- 2 ms -> 198 ms +- 5 ms: 1.07x slower (+7%) - unpickle_pure_python: 368 us +- 5 us -> 394 us +- 9 us: 1.07x slower (+7%) - sympy_expand: 426 ms +- 10 ms -> 452 ms +- 12 ms: 1.06x slower (+6%) - sympy_sum: 90.4 ms +- 0.6 ms -> 96.0 ms +- 1.0 ms: 1.06x slower (+6%) - regex_compile: 181 ms +- 7 ms -> 192 ms +- 7 ms: 1.06x slower (+6%) - scimark_lu: 173 ms +- 6 ms -> 182 ms +- 5 ms: 1.05x slower (+5%) - genshi_xml: 62.7 ms +- 0.8 ms -> 66.1 ms +- 0.8 ms: 1.05x slower (+5%) - pickle: 9.11 us +- 0.13 us -> 9.59 us +- 0.06 us: 1.05x slower (+5%) Faster (2): - unpack_sequence: 49.1 ns +- 0.7 ns -> 45.0 ns +- 1.3 ns: 1.09x faster (-8%) - scimark_sparse_mat_mult: 3.75 ms +- 0.05 ms -> 3.47 ms +- 0.05 ms: 1.08x faster (-8%) Benchmark hidden because not significant (29): 2to3, chaos, deltablue, dulwich_log, fannkuch, genshi_text, go, hexiom, html5lib, json_loads, logging_simple, meteor_contest, nqueens, pathlib, pidigits, python_startup, python_startup_no_site, regex_dna, regex_effbot, regex_v8, richards, scimark_fft, scimark_sor, spectral_norm, sqlite_synth, sympy_integrate, tornado_http, unpickle, unpickle_list Ignored benchmarks (5) of ../cpython-profile-tagged-off/base4.json: sqlalchemy_declarative, xml_etree_generate, xml_etree_iterparse, xml_etree_parse, xml_etree_process Ignored benchmarks (4) of fixedint5.json: xml_etree_pure_python_generate, xml_etree_pure_python_iterparse, xml_etree_pure_python_parse, xml_etree_pure_python_process

4 4

[capi-sig]tagged pointer fixed integer experiment
by Neil Schemenauer 16 Sep '18

16 Sep '18

This is an experiment to exercise the new C-API we are trying to design. I would like it to be able to support tagged pointers. This requires PyObject* to be treated as an opaque pointer. Note, I'm not suggesting that we should used tagged pointers in CPython. Instead, I just want to see if the API will make it possible to do it. The logic is that some Python implementations might want to use tagged pointers. They would also like to provide the same C-API as CPython. So, it would be real nice if we don't make things really hard for them. The patch adding the actual tagged pointer type is quite trivial. There are a number of git commits preceeding it in order to make it trivial. Coccinelle has been useful. Some of my semantic patches > @@ > expression E; > @@ > > -E->ob_type > +Py_TYPE(E) > @@ > expression E, F; > @@ > > -Py_TYPE(E) = F > +Py_SET_TYPE(E, F) Run like: spatch --sp-file ob_type.cocci <C source files> To speed things up, I used grep to narrow down the set of files to process. Here is the source code for the fixed int cpython. There is a new builtin 'fixedint'. It doesn't do much yet but at least doesn't immediately crash. https://github.com/nascheme/cpython/tree/tagged_int

8 21

[capi-sig]Adding an official (minimal) Cython-like tool.
by Eric Snow 09 Sep '18

09 Sep '18

Not that long ago Brett, Barry, and I were talking about how to get extension authors to move away from the C-API. Cython is the obvious choice, but it isn't an official tool nor does it necessarily make sense to make it one. Regardless, it would help all parties involved if there *were* an official tool that was part of CPython (i.e. in the repo). Cython could build on top of it and extension authors would be encouraged to use it or Cython. If such a thing makes sense, I figure we would follow the pattern set by asyncio (relative to twisted). -eric

6 12

[capi-sig]Re: No C API if it's not doable in Python
by Jeroen Demeyer 07 Sep '18

07 Sep '18

On 2018-09-07 14:47, Victor Stinner wrote: > Inside CPython, the core and builtin modules, micro-optimizations must > be used: abuse borrowed references, macros, access directly to all > fields of C structure, etc. > > But here I'm talking about the public C API used by third party extensions. Making a difference between "inside CPython" and "third party extensions" is a bad idea. Making such a difference would be problematic: 1. Complexity: we should not have two different C APIs. There should be just one, usable internally and externally. 2. Performance: if an optimization is important for CPython, then it's also important for third-party extensions. We don't want third-party code to be inherently slower than CPython itself. (this reminds me of PEP 580, regarding several internal optimizations which aren't available to third-party classes) Jeroen.

6 6

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 07 Sep '18

07 Sep '18

On 2018-09-04 23:50, Victor Stinner wrote: > I would like to design a new C API without borrow references. So > Py_TYPE() should go, replaced by something else. I don't see the problem with Py_TYPE(). Consider it an accessor macro, just a nice way to access to ob_type field.

12 38

[capi-sig]PyUnicode C API
by Victor Stinner 07 Sep '18

07 Sep '18

On 07.09.2018 10:22, Victor Stinner wrote: > > I'm in discussion with PyPy developers, and they reported different > > APIs which cause them troubles: > > (...) > > * almost all PyUnicode API functions have to go according to them. > > PyPy3 uses UTF-8 internally, CPython uses "compact string" (array of > > Py_UCS1, Py_UCS2 or Py_UCS4 depending on the string content). > > https://pythoncapi.readthedocs.io/bad_api.html#pypy-requests Le ven. 7 sept. 2018 à 10:33, M.-A. Lemburg <mal(a)egenix.com> a écrit : > I'm -1 on removing the PyUnicode APIs. We deliberately created a > useful and very complete C API for Unicode. > > The fact that PyPy chose to use a different internal representation > is not a good reason to remove APIs and have CPython extension take > the hit as a result. It would be better for PyPy rethink the > internal representation or create a shim API which translates > between the two worlds. > > Note that UTF-8 is not a good internal representation for Unicode > if you want fast indexing and slicing. This is why we are using > fixed code units to represent the Unicode strings. The PyUnicode C API is not only an issue for PyPy, it's also an issue for CPython. When the PEP 393 has been implemented, suddly, most of the PyUnicode API has been directly deprecated: all functions using the now legacy Py_UNICODE* type... Python 3.7 still has to support both the legacy Py_UNICODE* API and the new "compact string" API. It makes the CPython code base way more complex that it should be: any function accepting a string is supposed to call PyUnicode_Ready() and handle error properly. I would prefer to be able to remove the legacy PyUnicodeObject type, to only use compact strings everywhere. Let me elaborate what are good and bad functions for PyUnicode. Example of bad APIs: * PyUnicode_IS_COMPACT(): this API really rely on the *current* implementation * PyUnicode_2BYTE_DATA(): should only be used internally, there is no need to export it * PyUnicode_READ() * Py_UNICODE_strcmp(): use Py_UNICODE which is an implementation detail Good API: * PyUnicode_Concat(): C API for str + str * PyUnicode_Split() * PyUnicode_FindChar() Border line: * PyUnicode_IS_ASCII(op): it's a O(1) operation on CPython, but it can O(n) on other implementations (like PyPy which uses UTF-8). But we also added str.isascii() in Python 3.7.... * PyUnicode_READ_CHAR() * PyUnicode_CompareWithASCIIString(): the function name announces ASCII but decodes the byte string from Latin1 :-) Victor

3 8

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

capi-sig September 2018