I guess part of where I'm not sure I'm on solid ground is... is the pure-python version guaranteed to always exist and always be available under the name _Unpickler, or is that an implementation detail? I've been assuming that there was no such guarantee and any change would have to be clearly defined and ultimately available in both versions.
IIRC, it's not guaranteed, but it's generally considered best practice to have a pure Python version in the stdlib, even though that adds maintenance overhead. There are some things that can't be done in Python without C assistance, so it can't be a hard rule.
I'm basically just trying to push back against "this is, like eval, the keys to the kingdom and thus not worth hardening in any way at all".
And I'm basically trying to push back on the notion that this kind of discussion can be useful without talking about benefits (applications that can use the hardened pickle but not unhardened pickle) and threats (which define the "can use ... but not" phrase).
Sure - it'd have to be a new opcode at this point,
Why? The REDUCE opcode invokes load_reduce which ... oh heck, just post it:
def load_reduce(self): stack = self.stack args = stack.pop() func = stack[-1] stack[-1] = func(*args) dispatch[REDUCE] = load_reduce
So, why not this?
# in _pickle this variable needs to be exposed to Python call_restricting_callable_and_args = None def load_reduce(self): stack = self.stack args = stack.pop() func = stack[-1] if call_restricting_callable_and_args is None: stack[-1] = func(*args) else: stack[-1] = call_restricting_callable_and_args(func, args) dispatch[REDUCE] = load_reduce
which would allow raising an error, substituting a value, or calling func on args, as you suggested, but also allow substituting in args or even for func, and value substitution or editing after checking the value of func(*args). In C that would be as fast as you can get in the default case. Yes, there'd probably be pushback, even for that little overhead. But if you've got any good use cases, there'd be a shot, I think.
The problem for me is I don't know of any use cases for any of that flexibility beyond whitelisting func, which we already have.
I just think the wrong decision was made in the first place,
Which "first place", the earlier pickle that had a restricted mode, or the "modern" pickle which based on that experience removed restricted mode?