On 15 June 2013 22:41, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 15 Jun 2013 22:22:33 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
For custom allocators, it's useful to be able to *ensure* you can bypass CPython's small object allocator, rather than having to rely on it being bypassed for allocations above a certain size.
Which custom allocators?
Those used by companies like Dropbox to speed up frequent allocations (look up their PyCon 2011 keynote). If we don't provide suitable APIs that we can still hook in debug mode, they'll bypass our infrastructure completely and we'll miss significant memory accesses.
As for the debug functions you added: PyMem_GetRawAllocators(), PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(), PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't you try to make that 2 or 3?
Faux simplicity that is achieved only by failing to model a complex problem domain correctly is a bad idea (if we were satisfied with that, we could stick with the status quo).
Actually, I'm sure almost everyone *is* satisfied with the status quo here (witness the total absence of bug reports on the matter). Victor's patch addresses a rare concern compared to the common use cases of CPython.
Indeed, but they're use cases I care about, Victor cares about, Kristjan cares about, Greg cares about. It's OK that you don't care about them, just as 99% of the Python programmers on the planet won't care about PEP 432 or the various arcane metaclass changes we've made over the years. issue 3329 (the one where Victor implemented this) was actually filed by the folks working on the Symbian port. The last comment on that issue before Victor restarted was from you, in reply to someone asking if we had implemented it yet.
And I'm not even sure what "faux simplicity" you are talking about. What is the supposed complexity that this API is supposed to address?
The fact that there is more to the world than x86/x86_64 and the very simplistic C memory model. Python is growing more popular in non-traditional execution environments, and we finally have someone (Victor) interested in doing the work to support them properly. That should be celebrated, not blocked because it isn't meaningful for the more common systems where the C memory model is fine.
Why do we need two different pairs of hook-setting functions, rather than letting each function set / get all hooks at once?
I've already said I don't think the raw allocators should be configurable at runtime. The other is because it's likely people will only want to replace the lower level allocators and leave the small object allocator alone. However, they should be able to completely replace the small object allocator if they want. Making the more common case more complicated to avoid adding a more appropriate two level API is the kind of thing I mean by "faux simplicity" - it's almost certainly going to be harder to use in practice, so trading multiple functions for fewer functions each taking more parameters isn't actually a win.
And why the private API functions for setting arena allocators?
Because they're in a different compilation unit...
Memory allocation APIs are a fundamental part of the C API that many extension writers have to understand and deal with. I'm opposed to gratuitous complication when the use cases are not compelling.
That's a documentation problem. C extension authors shouldn't be touching these, and most people embedding CPython shouldn't be touching them either. They're the C level equivalent of metaclasses: if you're not sure you need them, you don't need them. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia