Mailman 3 December 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau Oct. 16, 2023

Oct. 16, 2023

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to … [View More]prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian [View Less]

11 14

Python multithreading without the GIL
by Sam Gross July 31, 2023

July 31, 2023

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem,… [View More]

24 48

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (… [View More]flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) [View Less]

7 17

[RELEASE] Python 3.11.1, 3.10.9, 3.9.16, 3.8.16, 3.7.16, and 3.12.0 alpha 3 are now available
by Łukasz Langa Feb. 18, 2023

Feb. 18, 2023

Greetings! We bring you a slew of releases this fine Saint Nicholas / Sinterklaas day. Six simultaneous releases has got to be some record. There’s one more record we broke this time, you’ll see below. In any case, updating is recommended due to security content: 3.7 - 3.12: gh-98739 <https://github.com/python/cpython/issues/98739>: Updated bundled libexpat to 2.5.0 to fix CVE-2022-43680 <https://nvd.nist.gov/vuln/detail/CVE-2022-43680> (heap use-after-free). 3.7 - 3.12: gh-98433 … [View More]

3 3

PEP 638: Syntactic macros
by Mark Shannon Feb. 5, 2023

Feb. 5, 2023

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

Switching to Discourse
by Petr Viktorin Jan. 26, 2023

Jan. 26, 2023

Hello, Currently development discussions are split between multiple communication channels, for example: - python-dev and discuss.python.org for design discussions, - GitHub Issues and Pull Requests for specific changes, - IRC, Discord and private chats for real-time discussions, - Topic-specific channels like typing-sig. While most of these serve different needs, there is too much overlap between python-dev and discuss.python.org. It seems that for most people, this situation is worse than … [View More]sticking to either one platform – even if we don't go with that person's favorite. The discuss.python.org experiment has been going on for quite a while, and while the platform is not without its issues, we consider it a success. The Core Development category is busier than python-dev. According to staff, discuss.python.org is much easier to moderate.. If you're following python-dev but not discuss.python.org, you're missing out. The Steering Council would like to switch from python-dev to discuss.python.org. Practically, this means: - Moving the required PEP announcements to discuss.python.org - Moving discuss.python.org up in the devguide communications page (https://devguide.python.org/communication/) - And that's it? I imagine that the mailing list will stay around for continuing past discussion threads and for announcements, eventually switching to auto-reject incoming messages with a pointer to discuss.python.org. To be clear, discuss.python.org allows editing posts, which is frankly handy for typos and clarifications. Editing alone should not be used for adding new info -- we should cultivate a culture of being friendly to mail users & notification watchers. This probably bears repeating in a few places. We're aware not everyone wants to use the discuss.python.org website, but there are some ways to avoid it: - For new PEPs, you can point your RSS client to https://www.python.org/dev/peps/peps.rss – it's not e-mail, but many email clients have RSS support. You can also watch the Steering Council issues on GitHub (https://github.com/python/steering-council/issues/) for important questions and discussions. - You can use discuss.python.org's “mailing list mode” (which subscribes you to all new posts), possibly with filtering and/or categorizing messages locally. However, we would like to know if this will pose an undue burden to anyone, if there are workflows or usage problems that we are not aware of. As mentioned, this is something the Steering Council thinks is a good idea, but we want to make sure we're aware of all the impact when we make the final decision. – Petr, on behalf of the Steering Council [View Less]

43 114

Re: PEP 701 – Syntactic formalization of f-strings
by Rob Cliffe Dec. 22, 2022

Dec. 22, 2022

Great stuff! 😁 Rob Cliffe On 19/12/2022 17:59, Pablo Galindo Salgado wrote: > > Hi everyone, > > I am very excited to share with you a PEP thatBatuhan Taskaya, > Lysandros Nikolaou and myself have been working on recently:PEP 701 - > PEP 701 – Syntactic formalization of f-strings > <https://peps.python.org/pep-0701/>. > > We believe this will be a great improvement in both the > maintainability of CPython and the usability of f-strings. > > We … [View More]

1 0

Feature Request: Adding Way to Annotate Class Variables Distinct from Instance Variables
by sakaic2003＠gmail.com Dec. 22, 2022

Dec. 22, 2022

Hello folks, I am Chihiro, a.k.a. Frodo821, and this is my first post to this group. I searched for a way to annotate both class variables and instance variables with different types and concluded that there is currently no way to do this. For example, what I want to: ``` class Foo: variable_both_class_and_instance = Field(desc="descriptive string") def __init__(self, value: int): self.variable_both_class_and_instance = value assert isinstance(Foo.… [View More]

4 3

PEP 701 – Syntactic formalization of f-strings
by Pablo Galindo Salgado Dec. 21, 2022

Dec. 21, 2022

Hi everyone, I am very excited to share with you a PEP that Batuhan Taskaya, Lysandros Nikolaou and myself have been working on recently: PEP 701 - PEP 701 – Syntactic formalization of f-strings <https://peps.python.org/pep-0701/>. We believe this will be a great improvement in both the maintainability of CPython and the usability of f-strings. We look forward to hearing what you think about this and to get your feedback! *Here is a TLDR for your convenience:* - The PEP proposes a … [View More]

1 1

computersmarketing
by jugnu7271＠gmail.com Dec. 20, 2022

Dec. 20, 2022

I am jugnu25 working for computers marketing PR consultant. With extra than 6 years experience in PR and Digital Industry, helping groups to obtain dreams through streamilining the technique. https://www.computersmarketing.com/

2 1

problem with abandonned lib pyliblo
by Mathieu Picot Dec. 20, 2022

Dec. 20, 2022

Hi. There is a problem with the lib pyliblo, the source code of the lib is here: https://github.com/dsacre/pyliblo The problem is that the lib is now incompatible with python3.11 because it uses inspect.getargspec function which does not exist anymore. This is very easy to fix, the main problem is that the author Dominic Sacré does not answer anymore to any message (PR, issues, email), so there is no hope he will fix the problem. I am ready to maintain this lib myself, I could simply fork it … [View More]

2 1

probably nothing but just in case.... 503 Service Unavailable error
by wonsuk yang Dec. 16, 2022

Dec. 16, 2022

hi! thank you for amazing community! the python community is the light and the salt for me! the site https://buildbot.python.org/all/#/grid from https://www.python.org/dev/buildbot/ shows the error of "503 Service Unavailable; No server is available to handle this request." i reckon the site might be deprecated yet report this just in case sorry to bother if it is already known! hope the best w. yang

2 2

Python Launcher not Installed on Windows Machine
by Margo Hoover Dec. 15, 2022

Dec. 15, 2022

Dear Jonathan and Brian, I have python and pip installed on my Windows machine. However, I do not seem to have the python launcher installed (if that is even possible). I am a very beginner in Python. Could you explain how I should 'download' the python installer, or if I should even do that. What would be a next step? Thank you once again, Sincerely, Margaux Hoover -- Margaux Hoover Xtrepid Design

3 2

Re: A proposal to modify `None` so that it hashes to a constant
by Rob Cliffe Dec. 11, 2022

Dec. 11, 2022

Wild suggestion: Make None.__hash__ writable. E.g. None.__hash__ = lambda : 0 # Currently raises AttributeError: 'NoneType' object attribute '__hash__' is read-only Best wishes Rob Cliffe On 01/12/2022 11:02, Oscar Benjamin wrote: > On Thu, 1 Dec 2022 at 06:56, Chris Angelico <rosuav(a)gmail.com> wrote: >> On Thu, 1 Dec 2022 at 17:26, Yoni Lavi <yoni.lavi.p(a)gmail.com> wrote: >>> So it's not like it's even possible to … [View More]

3 3

Re: A proposal to modify `None` so that it hashes to a constant
by Rob Cliffe Dec. 9, 2022

Dec. 9, 2022

You're right of course. Oh well, it *was* a wild idea.😁 Rob Cliffe On 04/12/2 On 04/12/2022 18:16, Chris Angelico wrote: > On Mon, 5 Dec 2022 at 05:11, Rob Cliffe via Python-Dev > <python-dev(a)python.org> wrote: >> Wild suggestion: >> Make None.__hash__ writable. >> E.g. >> None.__hash__ = lambda : 0 # Currently raises AttributeError: >> 'NoneType' object attribute '__hash__' is read-only > Hashes have to be stable. If you … [View More]

1 0

A proposal to modify `None` so that it hashes to a constant
by Yoni Lavi Dec. 3, 2022

Dec. 3, 2022

I wrote a doc stating my case here: https://docs.google.com/document/d/1et5x5HckTJhUQsz2lcC1avQrgDufXFnHMin7GlI… Briefly, 1. The main motivation for it is to allow users to get a predictable result on a given input (for programs that are doing pure compute, in domains like operations research / compilation), any time they run their program. Having stable repro is important for debugging. Notebooks with statistical analysis are another similar case where this is needed: you might want other … [View More]

11 29

Re: A proposal to modify `None` so that it hashes to a constant
by Rob Cliffe Dec. 3, 2022

Dec. 3, 2022

Thank you for this very clear analysis, Oscar. It seems to me that this strengthens the OP's case. I am curious as to whether others agree. Best wishes Rob Cliffe On 30/11/2022 13:35, Oscar Benjamin wrote: > On Tue, 29 Nov 2022 at 23:46, Steven D'Aprano <steve(a)pearwood.info> wrote: >> On Tue, Nov 29, 2022 at 08:51:09PM -0000, Yoni Lavi wrote: >> >>> It does make your argument invalid though, since it's based on this >>> assumption that I was … [View More]asking for a requirement on iteration order >>> (e.g. like dict's iteration order = insertion order guarantee), which >>> is not the case. >> Yoni, I think this answer is disingenious. > I don't think it is disingenuous. There are just a lot of people > talking past each other and not quite understanding what each person > means because there is confusion about even the intended meaning of > terms like "deterministic". I will expand here with enough detail that > we should hopefully be able to avoid misunderstanding each other. > > There are probably other places where you could find mentions of this > in the docs but I just took a quick look in the Python 3.5 docs > (before hash randomisation) to find this mention of dictionary > iteration order: > https://docs.python.org/3.5/library/stdtypes.html#dictionary-view-objects > > What it says is > """ > Keys and values are iterated over in an arbitrary order which is > non-random, varies across Python implementations, and depends on the > dictionary’s history of insertions and deletions. > """ > The key point is the use of the term "non-random" which here is > intended to mean that although no particular ordering is guaranteed > you can expect to rerun the same program and get the same result > deterministically. A different version or implementation of Python > might give a different order but rerunning the same program twice > without changing anything should give the same result even if that > result depends in some way on the iteration order of some > dictionaries. I can't immediately find a similar statement about sets > but in practice the same behaviour applied to sets as well. Note > carefully that it is this *narrow* form of determinism that Yoni is > interested in. > > Of course there are some caveats to this and the obvious one is that > this statement does not apply if there are some objects that use > identity based hashing so this is not deterministic: > > class A: > def __init__(self, data): > self.data = data > def __repr__(self): > return 'A(%s)' % self.data > > a1 = A(1) > a2 = A(2) > > for a in {a1, a2}: > print(a) > > Running this gives: > > $ python3.5 t.py > A(2) > A(1) > $ python3.5 t.py > A(1) > A(2) > > On the other hand if all of the hashes themselves are deterministic > then the program as a whole will be as well so this is deterministic: > > class A: > def __init__(self, data): > self.data = data > def __repr__(self): > return 'A(%s)' % self.data > def __hash__(self): > return hash(self.data) > def __eq__(self): > return self.data == other.data > > a1 = A(1) > a2 = A(2) > > for a in {a1, a2}: > print(a) > > $ python3.5 t.py > A(1) > A(2) > $ python3.5 t.py > A(1) > A(2) > > So we have two classes of hashable objects: > > 1. Those with deterministic hash > 2. Those with non-deterministic hash > > A program that avoids depending on the iteration order of sets or > dicts containing objects with non-deterministic hash could be > deterministic. It is not the case that the program would depend on the > iteration order for its *correctness* but just that the behaviour of > the program is *reproducible* which is useful in various ways e.g.: > > - You could say to someone else "run this code with CPython 3.5 and > you should be able to reproduce exactly what I see when I run the > program". It is common practice e.g. in scientific programming to > record things like random seeds so that someone else can precisely > reproduce the results shown in a paper or some other work and this in > general requires that it is at least possible to make everything > deterministic. > > - When debugging it is useful to be able to reproduce an error > condition precisely. Debugging non-deterministic failures can be > extremely difficult. In the same way that you might want to reproduce > correctly functioning code it is also very useful to be able to > reproduce bugs. > > I can list more examples but really it shouldn't be necessary to > justify from first principles why determinism in programming is > usually a good thing. There can be reasons sometimes why determinism > is undesired or cannot or should not be guaranteed. It should not be > controversial though to say that all things being equal determinism is > usually a desirable feature and should be preferred by default. I > don't think that the 3.5 docs I quoted above used the words > "non-random" casually: it was an intended feature and people were > aware that breaking that behaviour would be problematic in many > situations. > > Of course in Python 3.6 this determinism was broken with the > introduction of hash randomisation for strings. It was considered that > for security purposes it would be better to have some internal > non-deterministic behaviour to guard against attackers. Specifically > the hashes of three types (str, bytes and datetime) were made > non-deterministic between subsequent CPython processes. The effect was > not only to change purely internal state though but also the > observable iteration order of dicts and sets which became > non-deterministic from one run of CPython to another. It was > anticipated at the time that this might be problematic in some > situations (it certainly was!) and so an environment variable > PYTHONHASHSEED was introduced in order to restore determinism for > cases that needed it. > > So now if you want to have reproducible behaviour with Python 3.6+ you > also need to fix PYTHONHASHSEED as well as avoiding the use of other > types of non-deterministically hashable objects in sets and dicts. > This is something that I have personally used mainly for the > reproducibility of rare bugs e.g. "to reproduce this you should run > the following Python code using commit abc123 under CPython 3.6 and > with PYTHONHASHSEED=1234". > > Subsequently in Python 3.7 dict iteration order was changed to make it > always deterministic by having it not depend on the hash values at > all, with the order depending on the order of insertions into the dict > instead. This introduced a *stronger* guarantee of determinism: now > the ordering could be expected to be reproducible even with different > versions and implementations of Python. For many this seemed to have > resolved the problems of undefined, implementation-defined etc > ordering. However this only applied to dicts and not sets and as of > Python 3.12 any issues about deterministic ordering still remain > wherever sets are used. > > The introduction of hash randomisation means that since Python 3.6 > there are now three classes of hashable objects: > > 1. Objects with deterministic hash (int etc) > 2. Objects with non-deterministic hash that can be controlled with > PYTHONHASHSEED (bytes, str and datetime). > 3. Objects with non-deterministic hash that cannot be controlled > (id-based hashing). > > The question in this thread and others is which of these three classes > None should belong to. Although None is just one particular value it > is a very commonly used value and its non-deterministic hash can creep > through to affect the hash of larger data structures that use > recursive hash calls (e.g. a tuple of tuples of ... that somewhere > contains None). Also certain situations such as a type hint like > Optional[T] as referred to by the OP necessarily use None: > > $ python -c 'from typing import Optional; print(hash(Optional[int]))' > -7631587549837930667 > $ python -c 'from typing import Optional; print(hash(Optional[int]))' > -6488475102642892491 > > Somehow this affects frozen dataclasses but I haven't used those > myself so I won't demonstrate how to reproduce the problem with them. > > Here is a survey of types from builtins: > > NoneType bool bytearray bytes complex dict ellipsis enumerate filter > float frozenset int list map memoryview object range reversed set > slice str tuple type zip > > We can divide these into the three classes described above plus the > non-hashable types (I haven't checked this in the code, but just > experimenting with calling hash): > > 1. Deterministic hash: > bool, complex, float, frozenset, int, range, tuple > > 2. Hash depends on hash seed: > str, bytes > > 3. Hash depends on id: > NoneType, ellipsis, enumerate, filter, memoryview, object, reversed, zip > > 4. Non-hashable: > bytearray, dict, list, set, slice > > The question here is whether None belongs in class 3 or class 1. To me > it seems clear that there is no advantage in having None be in class 3 > except perhaps to save a few simple lines of code: > https://github.com/python/cpython/pull/99541/files > > There is however a worthwhile advantage in having None be in class 1. > If None had a deterministic hash then tuples, frozensets etc > consisting of objects with None as well as other objects with > deterministic hash could all have deterministic hash. The behaviour of > iteration order for sets would then be deterministic in the *narrow* > sense that is referred to by the Python 3.5 docs above. > > Some have argued that the fact that some types have a seed dependent > hash implies that None should not have a deterministic hash but this > does not follow. It was known at the time that str and bytes were > moved from class 1 to class 2 that it would be problematic to do so > which is precisely why PYTHONHASHSEED was introduced. However > PYTHONHASHSEED does not help here because None is not even in class 2 > but rather class 3. > > If I was going to label any claim made by anyone in these threads as > disingenuous then it would be the claim that None is somehow not > "special". Firstly many types have deterministic hash so it isn't > really that much of a special property. Secondly None is *clearly* a > special value that is used everywhere! When I open the CPython > interpreter there are already thousands of references to None before I > have even done anything: > > >>> import sys > >>> sys.getrefcount(None) > 4429 > > The motivation for this and other threads is to bring determinism in > the *narrow* sense. Others (including me) have made references to > other kinds of determinism that have derailed the threads by > misunderstanding exactly what Yoni is referring to. The *stronger* > sense of determinism would be useful if possible but it is not the > intended topic of these threads. > > -- > Oscar > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/73KMUPB… > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

4 5

RFC: expose ssl internals for use by ctypes/cffi
by christer＠weinigel.se Dec. 1, 2022

Dec. 1, 2022

Hi all, for quite some time I've been working on a Python implementation of a protocol called NTS which requires access to an API in OpenSSL which is not provided by the Python ssl module. I added a patch for that which unfortunately for me the maintainer did not want to accept. Some comments were made of a possible future way to give more generic access to the openssl library via ctypes/cffi but I have been unable to find more information about that. I was home sick last week and decided … [View More]to take a shot at it and have built something that I feel is a bit ugly but does seem to work. I'd like to some feedback on this approach. My patches can be found on github, based on the Python 3.11 tag: https://github.com/python/cpython/compare/3.11...wingel:cpython:main Here's a short description of each patch on this branch: "bpo-37952: SSL: add support for export_keying_material" is my old patch which adds the method I need to the ssl library just for reference. The other commits add the necessary infrastructure with some example code. These commits are not ready for submission but hopefully they show what I have in mind. "Add CRYPTO_DLL_PATH and SSL_DLL_PATH to the _ssl module. " This commit adds two constants to the "_ssl" C module with the paths to libcrypto and libssl respectively. On Linux dladdr and on Windows GetModuleHandle/GetModuleFilename are used on a symbol in each library to find the path to the corresponding DLL. I've verified that this works Debian Bulleye and on Windows 10 with Visual Studio 2017. I don't own a Mac so I haven't been able to test this on macOS, but I believe dladdr is available on modern macOS so it might work out of the box. With the paths it's possible to use ctypes or cffi get a handle to these libraries. "Add API to get the address of the SSL structure" then adds an API to an SSLSocket which returns the address of the corresponding "SSL" C structure. This address can be used by ctypes/cffi. One would probably want to expose SSL_CTX, SSL_SESSION and BIO too but I started with just SSL since that's what my code needs right now. "Add a small test program" is a small test program that uses the infrastructure from the two above commits to call C functions in libssl/libcrypto using both ctypes and cffi. It's a bit ugly but hopefully it's not too hard to understand. "Example of how to extend the ssl library using ctypes" is an example of how a Python module that extends the SSL library using ctypes could look. First get a handle to libssl using ctypes, set up ctypes with the correct API for the export_keying_material function, wrap it in a more Pythonic function and then extend SSLSocket with the new function. A simplified version looks like this: import ssl, ctypes ssl_lib = ctypes.CDLL(ssl._ssl.SSL_DLL_PATH) ssl_lib.SSL_export_keying_material.argtypes = ( ctypes.c_void_p, # SSL pointer ctypes.c_void_p, ctypes.c_size_t, # out pointer, out length ctypes.c_void_p, ctypes.c_size_t, # label buffer, label length ctypes.c_void_p, ctypes.c_size_t, # context, context length ctypes.c_int) # use context flag ssl_lib.SSL_export_keying_material.restype = ctypes.c_int def SSL_export_keying_material(self, label, key_len, context = None): c_key = ctypes.create_string_buffer(key_len) c_label = ctypes.create_string_buffer(len(label)) c_context = ctypes.create_string_buffer(context, len(context)) if ssl_lib.SSL_export_keying_material( self._sslobj.get_internal_addr(), c_key, key_len, c_label, len(label), c_context, len(context), 1); return bytes(c_key) ssl.SSLSocket.export_keying_material = SSL_export_keying_material There's a final commit "Expose more OPENSSL_ variables" which add some more constants to the ssl module which expose the cflags and build information from OpenSSL. This patch is not really necessary, but it might be a good idea to compare these constants with the corresponding constants retrieved using ctypes/cffi to make sure that exactly the same version of the openssl library is used. Does this seem like a good idea? As I said, I feel that it is a bit ugly, but it does mean that if someone wants to use some SSL_really_obscure_function in libcrypto or libssl they can do that without having to rebuild all of CPython themselves. Or if they want to integrate with some other C library that wants a raw pointer to a SSL socket. Hopefully this would reduce the burden on the ssl module maintainers a bit. Anyway, if you think this is a good approach I could clean up my patches, add support for SSL_CTX/SSL_SESSION/BIO, document all of this and make it into a proper pull request. /Christer [View Less]

5 4