Mailman 3 - Python-Dev

computersmarketing
by jugnu7271＠gmail.com Dec. 20, 2022

Dec. 20, 2022

I am jugnu25 working for computers marketing PR consultant. With extra than 6 years experience in PR and Digital Industry, helping groups to obtain dreams through streamilining the technique. https://www.computersmarketing.com/

2 1

problem with abandonned lib pyliblo
by Mathieu Picot Dec. 20, 2022

Dec. 20, 2022

Hi. There is a problem with the lib pyliblo, the source code of the lib is here: https://github.com/dsacre/pyliblo The problem is that the lib is now incompatible with python3.11 because it uses inspect.getargspec function which does not exist anymore. This is very easy to fix, the main problem is that the author Dominic Sacré does not answer anymore to any message (PR, issues, email), so there is no hope he will fix the problem. I am ready to maintain this lib myself, I could simply fork it and rename the lib to pass it to pip list, but all programs using it will have to rename the lib in their requirements. Is there a sort of protocol to do when this happens, in order to replace the source package with a new version ? thanks for your help Mathieu

2 1

probably nothing but just in case.... 503 Service Unavailable error
by wonsuk yang Dec. 16, 2022

Dec. 16, 2022

hi! thank you for amazing community! the python community is the light and the salt for me! the site https://buildbot.python.org/all/#/grid from https://www.python.org/dev/buildbot/ shows the error of "503 Service Unavailable; No server is available to handle this request." i reckon the site might be deprecated yet report this just in case sorry to bother if it is already known! hope the best w. yang

2 2

Python Launcher not Installed on Windows Machine
by Margo Hoover Dec. 15, 2022

Dec. 15, 2022

Dear Jonathan and Brian, I have python and pip installed on my Windows machine. However, I do not seem to have the python launcher installed (if that is even possible). I am a very beginner in Python. Could you explain how I should 'download' the python installer, or if I should even do that. What would be a next step? Thank you once again, Sincerely, Margaux Hoover -- Margaux Hoover Xtrepid Design

3 2

Re: A proposal to modify `None` so that it hashes to a constant
by Rob Cliffe Dec. 11, 2022

Dec. 11, 2022

Wild suggestion: Make None.__hash__ writable. E.g. None.__hash__ = lambda : 0 # Currently raises AttributeError: 'NoneType' object attribute '__hash__' is read-only Best wishes Rob Cliffe On 01/12/2022 11:02, Oscar Benjamin wrote: > On Thu, 1 Dec 2022 at 06:56, Chris Angelico <rosuav(a)gmail.com> wrote: >> On Thu, 1 Dec 2022 at 17:26, Yoni Lavi <yoni.lavi.p(a)gmail.com> wrote: >>> So it's not like it's even possible to require this generally for all objects. >> Well, I mean, in theory you could require that objects whose hash >> isn't otherwise defined get given the hash of zero. That doesn't >> violate any of the actual rules of hashes, but it does make those >> hashes quite suboptimal :) >> >> It's interesting how id() and hash() have opposite requirements (id >> must return a unique number among concurrently-existing objects, hash >> must return the same number among comparing-equal objects), yet a hash >> can be built on an id. > This also demonstrates a significant reason why None is special: it's > a singleton that only compares equal to itself. The reason for using > id for hash in other cases is to make different instances have > different hashes but there is only ever one instance of None. A > singleton class can have a hash function that matches identity based > equality without using id: any constant hash function will do. > > -- > Oscar > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MTTJJN2… > Code of Conduct: http://python.org/psf/codeofconduct/

3 3

Re: A proposal to modify `None` so that it hashes to a constant
by Rob Cliffe Dec. 9, 2022

Dec. 9, 2022

You're right of course. Oh well, it *was* a wild idea.😁 Rob Cliffe On 04/12/2 On 04/12/2022 18:16, Chris Angelico wrote: > On Mon, 5 Dec 2022 at 05:11, Rob Cliffe via Python-Dev > <python-dev(a)python.org> wrote: >> Wild suggestion: >> Make None.__hash__ writable. >> E.g. >> None.__hash__ = lambda : 0 # Currently raises AttributeError: >> 'NoneType' object attribute '__hash__' is read-only > Hashes have to be stable. If you change the hash of None after it's > been inserted into a dictionary, you'll get all kinds of entertaining > problems. > >>>> class X: > ... def __init__(self): self.hash = 0 > ... def __hash__(self): return self.hash > ... >>>> x = X() >>>> d = {x: "This is x"} >>>> x.hash = 1 >>>> for key in d: print(key, key in d) > ... > <__main__.X object at 0x7f2d07c6f1c0> False > > ChrisA > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OVVIKTG… > Code of Conduct: http://python.org/psf/codeofconduct/

1 0

A proposal to modify `None` so that it hashes to a constant
by Yoni Lavi Dec. 3, 2022

Dec. 3, 2022

I wrote a doc stating my case here: https://docs.google.com/document/d/1et5x5HckTJhUQsz2lcC1avQrgDufXFnHMin7GlI… Briefly, 1. The main motivation for it is to allow users to get a predictable result on a given input (for programs that are doing pure compute, in domains like operations research / compilation), any time they run their program. Having stable repro is important for debugging. Notebooks with statistical analysis are another similar case where this is needed: you might want other people to run your notebook and get the same result you did. 2. The reason the hash non-determinism of None matters in practice is that it can infect commonly used mapping key types, such as frozen dataclasses containing `Optional[int]` fields. 3. Non-determinism emerging from other value types like `str` can be disabled by the user using `PYTHONHASHSEED`, but there's no such protection against `None`. All it takes is for your program to compute a set somewhere with affected keys, and iterate on it - and determinism is lost. The need to modify None itself is caused by two factors - `Optional` being implemented effectively as `T | None` in Python as a strongly established practice - The fact that `__hash__` is an intrinsic property of a type in Python, the hashing function cannot be externally supplied to its builtin container types. So we have to modify the type None itself, rather than write some alternative hasher that we could use if we care about deterministic behavior across runs. This was debated at length over the forum and in discord. I also posted a PR for it, and it was closed, see: https://github.com/python/cpython/issues/99540 https://github.com/python/cpython/pull/99541 Asking for opinions, and to re-open the PR, provided there is enough support for such a change to take place.

11 29

Re: A proposal to modify `None` so that it hashes to a constant
by Rob Cliffe Dec. 3, 2022

Dec. 3, 2022

Thank you for this very clear analysis, Oscar. It seems to me that this strengthens the OP's case. I am curious as to whether others agree. Best wishes Rob Cliffe On 30/11/2022 13:35, Oscar Benjamin wrote: > On Tue, 29 Nov 2022 at 23:46, Steven D'Aprano <steve(a)pearwood.info> wrote: >> On Tue, Nov 29, 2022 at 08:51:09PM -0000, Yoni Lavi wrote: >> >>> It does make your argument invalid though, since it's based on this >>> assumption that I was asking for a requirement on iteration order >>> (e.g. like dict's iteration order = insertion order guarantee), which >>> is not the case. >> Yoni, I think this answer is disingenious. > I don't think it is disingenuous. There are just a lot of people > talking past each other and not quite understanding what each person > means because there is confusion about even the intended meaning of > terms like "deterministic". I will expand here with enough detail that > we should hopefully be able to avoid misunderstanding each other. > > There are probably other places where you could find mentions of this > in the docs but I just took a quick look in the Python 3.5 docs > (before hash randomisation) to find this mention of dictionary > iteration order: > https://docs.python.org/3.5/library/stdtypes.html#dictionary-view-objects > > What it says is > """ > Keys and values are iterated over in an arbitrary order which is > non-random, varies across Python implementations, and depends on the > dictionary’s history of insertions and deletions. > """ > The key point is the use of the term "non-random" which here is > intended to mean that although no particular ordering is guaranteed > you can expect to rerun the same program and get the same result > deterministically. A different version or implementation of Python > might give a different order but rerunning the same program twice > without changing anything should give the same result even if that > result depends in some way on the iteration order of some > dictionaries. I can't immediately find a similar statement about sets > but in practice the same behaviour applied to sets as well. Note > carefully that it is this *narrow* form of determinism that Yoni is > interested in. > > Of course there are some caveats to this and the obvious one is that > this statement does not apply if there are some objects that use > identity based hashing so this is not deterministic: > > class A: > def __init__(self, data): > self.data = data > def __repr__(self): > return 'A(%s)' % self.data > > a1 = A(1) > a2 = A(2) > > for a in {a1, a2}: > print(a) > > Running this gives: > > $ python3.5 t.py > A(2) > A(1) > $ python3.5 t.py > A(1) > A(2) > > On the other hand if all of the hashes themselves are deterministic > then the program as a whole will be as well so this is deterministic: > > class A: > def __init__(self, data): > self.data = data > def __repr__(self): > return 'A(%s)' % self.data > def __hash__(self): > return hash(self.data) > def __eq__(self): > return self.data == other.data > > a1 = A(1) > a2 = A(2) > > for a in {a1, a2}: > print(a) > > $ python3.5 t.py > A(1) > A(2) > $ python3.5 t.py > A(1) > A(2) > > So we have two classes of hashable objects: > > 1. Those with deterministic hash > 2. Those with non-deterministic hash > > A program that avoids depending on the iteration order of sets or > dicts containing objects with non-deterministic hash could be > deterministic. It is not the case that the program would depend on the > iteration order for its *correctness* but just that the behaviour of > the program is *reproducible* which is useful in various ways e.g.: > > - You could say to someone else "run this code with CPython 3.5 and > you should be able to reproduce exactly what I see when I run the > program". It is common practice e.g. in scientific programming to > record things like random seeds so that someone else can precisely > reproduce the results shown in a paper or some other work and this in > general requires that it is at least possible to make everything > deterministic. > > - When debugging it is useful to be able to reproduce an error > condition precisely. Debugging non-deterministic failures can be > extremely difficult. In the same way that you might want to reproduce > correctly functioning code it is also very useful to be able to > reproduce bugs. > > I can list more examples but really it shouldn't be necessary to > justify from first principles why determinism in programming is > usually a good thing. There can be reasons sometimes why determinism > is undesired or cannot or should not be guaranteed. It should not be > controversial though to say that all things being equal determinism is > usually a desirable feature and should be preferred by default. I > don't think that the 3.5 docs I quoted above used the words > "non-random" casually: it was an intended feature and people were > aware that breaking that behaviour would be problematic in many > situations. > > Of course in Python 3.6 this determinism was broken with the > introduction of hash randomisation for strings. It was considered that > for security purposes it would be better to have some internal > non-deterministic behaviour to guard against attackers. Specifically > the hashes of three types (str, bytes and datetime) were made > non-deterministic between subsequent CPython processes. The effect was > not only to change purely internal state though but also the > observable iteration order of dicts and sets which became > non-deterministic from one run of CPython to another. It was > anticipated at the time that this might be problematic in some > situations (it certainly was!) and so an environment variable > PYTHONHASHSEED was introduced in order to restore determinism for > cases that needed it. > > So now if you want to have reproducible behaviour with Python 3.6+ you > also need to fix PYTHONHASHSEED as well as avoiding the use of other > types of non-deterministically hashable objects in sets and dicts. > This is something that I have personally used mainly for the > reproducibility of rare bugs e.g. "to reproduce this you should run > the following Python code using commit abc123 under CPython 3.6 and > with PYTHONHASHSEED=1234". > > Subsequently in Python 3.7 dict iteration order was changed to make it > always deterministic by having it not depend on the hash values at > all, with the order depending on the order of insertions into the dict > instead. This introduced a *stronger* guarantee of determinism: now > the ordering could be expected to be reproducible even with different > versions and implementations of Python. For many this seemed to have > resolved the problems of undefined, implementation-defined etc > ordering. However this only applied to dicts and not sets and as of > Python 3.12 any issues about deterministic ordering still remain > wherever sets are used. > > The introduction of hash randomisation means that since Python 3.6 > there are now three classes of hashable objects: > > 1. Objects with deterministic hash (int etc) > 2. Objects with non-deterministic hash that can be controlled with > PYTHONHASHSEED (bytes, str and datetime). > 3. Objects with non-deterministic hash that cannot be controlled > (id-based hashing). > > The question in this thread and others is which of these three classes > None should belong to. Although None is just one particular value it > is a very commonly used value and its non-deterministic hash can creep > through to affect the hash of larger data structures that use > recursive hash calls (e.g. a tuple of tuples of ... that somewhere > contains None). Also certain situations such as a type hint like > Optional[T] as referred to by the OP necessarily use None: > > $ python -c 'from typing import Optional; print(hash(Optional[int]))' > -7631587549837930667 > $ python -c 'from typing import Optional; print(hash(Optional[int]))' > -6488475102642892491 > > Somehow this affects frozen dataclasses but I haven't used those > myself so I won't demonstrate how to reproduce the problem with them. > > Here is a survey of types from builtins: > > NoneType bool bytearray bytes complex dict ellipsis enumerate filter > float frozenset int list map memoryview object range reversed set > slice str tuple type zip > > We can divide these into the three classes described above plus the > non-hashable types (I haven't checked this in the code, but just > experimenting with calling hash): > > 1. Deterministic hash: > bool, complex, float, frozenset, int, range, tuple > > 2. Hash depends on hash seed: > str, bytes > > 3. Hash depends on id: > NoneType, ellipsis, enumerate, filter, memoryview, object, reversed, zip > > 4. Non-hashable: > bytearray, dict, list, set, slice > > The question here is whether None belongs in class 3 or class 1. To me > it seems clear that there is no advantage in having None be in class 3 > except perhaps to save a few simple lines of code: > https://github.com/python/cpython/pull/99541/files > > There is however a worthwhile advantage in having None be in class 1. > If None had a deterministic hash then tuples, frozensets etc > consisting of objects with None as well as other objects with > deterministic hash could all have deterministic hash. The behaviour of > iteration order for sets would then be deterministic in the *narrow* > sense that is referred to by the Python 3.5 docs above. > > Some have argued that the fact that some types have a seed dependent > hash implies that None should not have a deterministic hash but this > does not follow. It was known at the time that str and bytes were > moved from class 1 to class 2 that it would be problematic to do so > which is precisely why PYTHONHASHSEED was introduced. However > PYTHONHASHSEED does not help here because None is not even in class 2 > but rather class 3. > > If I was going to label any claim made by anyone in these threads as > disingenuous then it would be the claim that None is somehow not > "special". Firstly many types have deterministic hash so it isn't > really that much of a special property. Secondly None is *clearly* a > special value that is used everywhere! When I open the CPython > interpreter there are already thousands of references to None before I > have even done anything: > > >>> import sys > >>> sys.getrefcount(None) > 4429 > > The motivation for this and other threads is to bring determinism in > the *narrow* sense. Others (including me) have made references to > other kinds of determinism that have derailed the threads by > misunderstanding exactly what Yoni is referring to. The *stronger* > sense of determinism would be useful if possible but it is not the > intended topic of these threads. > > -- > Oscar > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/73KMUPB… > Code of Conduct: http://python.org/psf/codeofconduct/

4 5

RFC: expose ssl internals for use by ctypes/cffi
by christer＠weinigel.se Dec. 1, 2022

Dec. 1, 2022

Hi all, for quite some time I've been working on a Python implementation of a protocol called NTS which requires access to an API in OpenSSL which is not provided by the Python ssl module. I added a patch for that which unfortunately for me the maintainer did not want to accept. Some comments were made of a possible future way to give more generic access to the openssl library via ctypes/cffi but I have been unable to find more information about that. I was home sick last week and decided to take a shot at it and have built something that I feel is a bit ugly but does seem to work. I'd like to some feedback on this approach. My patches can be found on github, based on the Python 3.11 tag: https://github.com/python/cpython/compare/3.11...wingel:cpython:main Here's a short description of each patch on this branch: "bpo-37952: SSL: add support for export_keying_material" is my old patch which adds the method I need to the ssl library just for reference. The other commits add the necessary infrastructure with some example code. These commits are not ready for submission but hopefully they show what I have in mind. "Add CRYPTO_DLL_PATH and SSL_DLL_PATH to the _ssl module. " This commit adds two constants to the "_ssl" C module with the paths to libcrypto and libssl respectively. On Linux dladdr and on Windows GetModuleHandle/GetModuleFilename are used on a symbol in each library to find the path to the corresponding DLL. I've verified that this works Debian Bulleye and on Windows 10 with Visual Studio 2017. I don't own a Mac so I haven't been able to test this on macOS, but I believe dladdr is available on modern macOS so it might work out of the box. With the paths it's possible to use ctypes or cffi get a handle to these libraries. "Add API to get the address of the SSL structure" then adds an API to an SSLSocket which returns the address of the corresponding "SSL" C structure. This address can be used by ctypes/cffi. One would probably want to expose SSL_CTX, SSL_SESSION and BIO too but I started with just SSL since that's what my code needs right now. "Add a small test program" is a small test program that uses the infrastructure from the two above commits to call C functions in libssl/libcrypto using both ctypes and cffi. It's a bit ugly but hopefully it's not too hard to understand. "Example of how to extend the ssl library using ctypes" is an example of how a Python module that extends the SSL library using ctypes could look. First get a handle to libssl using ctypes, set up ctypes with the correct API for the export_keying_material function, wrap it in a more Pythonic function and then extend SSLSocket with the new function. A simplified version looks like this: import ssl, ctypes ssl_lib = ctypes.CDLL(ssl._ssl.SSL_DLL_PATH) ssl_lib.SSL_export_keying_material.argtypes = ( ctypes.c_void_p, # SSL pointer ctypes.c_void_p, ctypes.c_size_t, # out pointer, out length ctypes.c_void_p, ctypes.c_size_t, # label buffer, label length ctypes.c_void_p, ctypes.c_size_t, # context, context length ctypes.c_int) # use context flag ssl_lib.SSL_export_keying_material.restype = ctypes.c_int def SSL_export_keying_material(self, label, key_len, context = None): c_key = ctypes.create_string_buffer(key_len) c_label = ctypes.create_string_buffer(len(label)) c_context = ctypes.create_string_buffer(context, len(context)) if ssl_lib.SSL_export_keying_material( self._sslobj.get_internal_addr(), c_key, key_len, c_label, len(label), c_context, len(context), 1); return bytes(c_key) ssl.SSLSocket.export_keying_material = SSL_export_keying_material There's a final commit "Expose more OPENSSL_ variables" which add some more constants to the ssl module which expose the cflags and build information from OpenSSL. This patch is not really necessary, but it might be a good idea to compare these constants with the corresponding constants retrieved using ctypes/cffi to make sure that exactly the same version of the openssl library is used. Does this seem like a good idea? As I said, I feel that it is a bit ugly, but it does mean that if someone wants to use some SSL_really_obscure_function in libcrypto or libssl they can do that without having to rebuild all of CPython themselves. Or if they want to integrate with some other C library that wants a raw pointer to a SSL socket. Hopefully this would reduce the burden on the ssl module maintainers a bit. Anyway, if you think this is a good approach I could clean up my patches, add support for SSL_CTX/SSL_SESSION/BIO, document all of this and make it into a proper pull request. /Christer

5 4

Re: Python-Dev Digest, Vol 232, Issue 10
by Yoni Lavi Nov. 29, 2022

Nov. 29, 2022

> I can't see any, but then I couldn't see the security consequences of > predictable string hashes until they were pointed out to me. So it would > be really good to have some security experts comment on whether this is > safe or not. I can't either. I can point out that the complexity attack via hash collisions is not possible on None specifically because it is a singleton, so: 1. There may only be one None in a dict at most, 2. For composite types that depend on None and other things, like say a tuple of string and an optional int, they become as resistant to attack as the same types without None, in this case string and int. 3. Python only explicitly added this security (hash randomization) feature to string and bytes hashes, and if your composite key type depends on those types, it will still be protected regardless of what None does > Because this entire discussion is motivated by the OP who wants > consistent set order across multiple runs of his Python application. > That's what he needs; having None hash to a constant value is just a > means to that end. Not entirely. I do explain in my doc why there is a foundational reason why, once the choice was made to have None represent the `None` of Optional[T], it entered the family of types that you would expect to have deterministic behavior. And as the hashing function is intrinsic to types in Python, it is included in that. > Even if we grant None a constant hash, that still does not guarantee > consistent set order across runs. At best, we might get such consistent > order as an undocumented and changeable implementation detail, until we > change the implementation of hashing, or of sets, or of something > seemingly unrelated like address randomisation. ASLR will not cause any trouble so long as I keep objects with identity based hashing out of my sets. Or at least, any sets I later iterate on and run imperative code with side-effects under the loop. The possibility of disabling ASLR is not a reason to dismiss this change. No one guarantees a user of Python is in a position to make infosec related decisions on the computer system they are working on. Regarding the possibilities of hashes changing for the worse (in this regard) - sure. Anything is possible. Regarding sets - the set implementation is deterministic, and always has been (for decades). Yes, in theory it is possible that a new version or implementation of Python will make sets non-deterministic - shuffle themselves in the background independently from their history of operations, or what have you, and then my change loses all of its value. Like I said, anything is possible. But I think these last two points are essentially FUD. I made my proposal because I believe the FUD scenarios are strongly unlikely. (and even then, at worst we end up with a "practically useless" behavior on None, that can also be freely reverted along with such other changes anyway) On Tue, Nov 29, 2022 at 5:34 AM <python-dev-request(a)python.org> wrote: > Send Python-Dev mailing list submissions to > python-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman3/lists/python-dev.python.org/ > or, via email, send a message with subject or body 'help' to > python-dev-request(a)python.org > > You can reach the person managing the list at > python-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..."Today's Topics: > > 1. Re: A proposal to modify `None` so that it hashes to a constant > (Steven D'Aprano) > 2. Re: A proposal to modify `None` so that it hashes to a constant > (Oscar Benjamin) > 3. Re: A proposal to modify `None` so that it hashes to a constant > (Chris Angelico) > 4. Re: A proposal to modify `None` so that it hashes to a constant > (Steven D'Aprano) > > > > ---------- Forwarded message ---------- > From: "Steven D'Aprano" <steve(a)pearwood.info> > To: python-dev(a)python.org > Cc: > Bcc: > Date: Mon, 28 Nov 2022 15:35:44 +1100 > Subject: [Python-Dev] Re: A proposal to modify `None` so that it hashes to > a constant > On Tue, Nov 29, 2022 at 01:34:54PM +1300, Greg Ewing wrote: > > > I got the impression that there were some internal language reasons > > to want stable dicts, e.g. so that the class dict passed to __prepare__ > > preserves the order in which names are assigned in the class body. Are > > there any such use cases for stable sets? > > Some people wanted order preserving kwargs, I think for web frameworks. > There was even a discussion for a while about using OrderedDict for > kwargs and leaving dicts unordered. > > For me, the biggest advantage of preserving input order in dicts is that > it is now easier to write doctests using the dict's repr. It would be > nice to be able to do the same for sets, but not nice enough to justify > making them bigger or slower. > > > -- > Steve > > > > ---------- Forwarded message ---------- > From: Oscar Benjamin <oscar.j.benjamin(a)gmail.com> > To: > Cc: python-dev(a)python.org > Bcc: > Date: Tue, 29 Nov 2022 02:07:34 +0000 > Subject: [Python-Dev] Re: A proposal to modify `None` so that it hashes to > a constant > On Tue, 29 Nov 2022 at 01:33, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > On Mon, Nov 28, 2022 at 11:13:34PM +0000, Oscar Benjamin wrote: > > > On Mon, 28 Nov 2022 at 22:56, Brett Cannon <brett(a)python.org> wrote: > > > > As I understand it, we could make sets ordered, but only at the cost of > > space (much more memory) or time (slower) or both. > > > > I am sure that Guido is correct that **if** somebody comes up with a > > fast, efficient ordered set implementation that doesn't perform worse > > than the current implementation, we will happily swap to giving sets a > > predictable order, as we did with dicts. (Practicality beats purity -- > > even if sets are *philosophically* unordered, preserving input order is > > too useful to give up unless we gain something in return.) > > Let's split this into two separate questions: > > 1. Is it *innately* good that set order is non-deterministic? > 2. Are there some other reasons why it is good to choose a model that > implies *non-deterministic* set order? > > The answer to 1. is emphatically NO. In fact the question itself is > badly posed: why are we even asking about "set order" rather than the > benefits of determinism in general? If I want my code to be > deterministic then that's just something that I want regardless of > whether sets, dicts, floats etc are involved. > > As for point 2. the fact that sets are currently non-deterministic is > actually a relatively new thing in Python. Before hash-randomisation > set and dict order *was* deterministic but with an arbitrary order. > That was only changed because of a supposed security issue with hash > collisions. Prior to that it was well understood that determinism was > beneficial (honestly I don't understand why I have to state this point > explicitly: determinism is almost always best in our context). > > Please everyone don't confuse arbitrary order, implementation defined > order and non-deterministic order. There is no reason why sets in > Python need to have a *non-deterministic* order or at least why there > shouldn't be a way to control that. There is no performance penalty in > making the order *deterministic*. (If you think that there might be a > performance penalty then you haven't understood the suggestion!) > > > > It would be useful to have a straight-forward way to sort a set into a > > > deterministic ordering but no such feature exists after the Py3K > > > changes (sorted used to do this in Python 2.x). > > > > `sorted()` works fine on homogeneous sets. It is only heterogeneous sets > > that are a problem, and in practice, that usually means None mixed in > > with some other type. > > That is of course precisely the context for this thread! > > -- > Oscar > > > > ---------- Forwarded message ---------- > From: Chris Angelico <rosuav(a)gmail.com> > To: python-dev(a)python.org > Cc: > Bcc: > Date: Tue, 29 Nov 2022 13:20:30 +1100 > Subject: [Python-Dev] Re: A proposal to modify `None` so that it hashes to > a constant > On Tue, 29 Nov 2022 at 13:12, Oscar Benjamin <oscar.j.benjamin(a)gmail.com> > wrote: > > As for point 2. the fact that sets are currently non-deterministic is > > actually a relatively new thing in Python. Before hash-randomisation > > set and dict order *was* deterministic but with an arbitrary order. > > That was only changed because of a supposed security issue with hash > > collisions. Prior to that it was well understood that determinism was > > beneficial (honestly I don't understand why I have to state this point > > explicitly: determinism is almost always best in our context). > > To clarify: The hash collision attack is a very real one, but specific > to dictionaries of string keys, since there are quite a few ways for > an attacker to send a string that gets automatically parsed into such > a dictionary (eg web app frameworks where the request parameters are > made available as a dictionary). But since that attack surface is *so* > specific, randomization of non-string hashes is unimportant. > > ChrisA > > > > ---------- Forwarded message ---------- > From: "Steven D'Aprano" <steve(a)pearwood.info> > To: python-dev(a)python.org > Cc: > Bcc: > Date: Mon, 28 Nov 2022 17:19:22 +1100 > Subject: [Python-Dev] Re: A proposal to modify `None` so that it hashes to > a constant > On Tue, Nov 29, 2022 at 02:07:34AM +0000, Oscar Benjamin wrote: > > > Let's split this into two separate questions: > > Let's not. Your first question about non-deterministic set order being > "innately good" is a straw man: as we've already discussed, set order is > not non-deterministic (except in the informal sense of "hard to > predict") and I don't think anyone is arguing in favour of keeping set > order unpredictable even if there are faster, more compact, simpler > implementations which preserve order. > > Talking about determinism is just muddying the waters. Sets are > deterministic: their order is determinied by the implementation, the set > history, and potentially any environmental sources of entropy used in > address randomisation. Deterministic does not mean predictable. > > If we want to get this discussion onto a more useful path, we should > start with a security question: > > The hash of None changes because of address randomisation. Address > randomisation is enabled as a security measure. Are there any security > consequences of giving None a constant hash? > > I can't see any, but then I couldn't see the security consequences of > predictable string hashes until they were pointed out to me. So it would > be really good to have some security experts comment on whether this is > safe or not. > > > > why are we even asking about "set order" rather than the > > benefits of determinism in general? > > Because this entire discussion is motivated by the OP who wants > consistent set order across multiple runs of his Python application. > That's what he needs; having None hash to a constant value is just a > means to that end. > > His sets contain objects whose hashes depend on `Optional[int]`, which > means sometimes they include None, and when he runs under an interpreter > built with address randomisation, the hash of None can change. > > Even if we grant None a constant hash, that still does not guarantee > consistent set order across runs. At best, we might get such consistent > order as an undocumented and changeable implementation detail, until we > change the implementation of hashing, or of sets, or of something > seemingly unrelated like address randomisation. > > > -- > Steve > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ >

1 0