Mailman 3 March 2016 - Python-ideas

MySQL GUI
by isaac tetteh March 29, 2016

March 29, 2016

Hello, I was looking at creating a gui in Python to connect to a mysql database to do SIDU. What will be the best GUI package to use. Thanks :) Sent from my iPhone

2 2

Re: [Python-ideas] `to_file()` method for strings
by Nick Coghlan March 29, 2016

March 29, 2016

On 23 March 2016 at 15:44, Nick Eubank <nickeubank(a)gmail.com> wrote: > True -- my feeling is that if we're ok with defaults in a pathlib method, > why not put them in a string method? pathlib is explicitly about filesytem management and manipulation, while strings don't inherently have anything to do with filesystems, and only a little bit to do with serialisation (via str.encode). Having some lower level convenience functions in io may make sense (specifically io.write_bytes(… [View More]

8 11

Re: [Python-ideas] `to_file()` method for strings
by Michel Desmoulin March 28, 2016

March 28, 2016

Woops, sent it to only one person. Le 28/03/2016 19:52, Michel Desmoulin a écrit : > > > Le 28/03/2016 19:24, Alexander Belopolsky a écrit : >> >> On Mon, Mar 28, 2016 at 12:40 PM, Chris Barker <chris.barker(a)noaa.gov >> <mailto:chris.barker@noaa.gov>> wrote: >> >> but: >> >> data = open("foo.txt", "r", close_after_use=True).read() >> >> and >> >> infile = open("foo.txt", "r", … [View More]

1 0

PEP 484 evolution
by Guido van Rossum March 27, 2016

March 27, 2016

I have a few different things I'd like to add to PEP 484. Should I start this here or on pyrhon-dev? The hope is to get these into 3.5.2. When is that going out? --Guido (mobile)

14 35

Type hints for text/binary data in Python 2+3 code
by Andrey Vlasovskikh March 26, 2016

March 26, 2016

With the addition of the comment-based syntax [3] for Python 2.7 + 3 in PEP 0484 having a Python 2/3 compatible way of adding type hints for text and binary values becomes important. Following the issue #1141 at the Mypy GitHub site [1], I've came up with a draft proposal based on those ideas that I'd like to discuss here. # Abstract This proposal contains recommendations on how to annotate text/binary data in newly added PEP 0484 comment-based type hints in order to make them Python 2/3 … [View More]compatible when the single-source approach to porting from 2 to 3 is used. It introduces a new type `typing.Text` that represents text data in both Python 2 and 3, deprecates `str -> unicode` promotion used in type checkers, suggests an approach for type checkers to find implicit conversion errors by tracking ASCII text/binary values, recommends that type checkers should warn about `unicode` in the 2+3 mode. # Rationale With the addition of the comment-based syntax for Python 2.7 + 3 having a Python 2/3 compatible way of annotating types of text and binary values becomes important. Currently having a single-source code base is the main approach to 2/3 compatibility, so it is highly desirable to have 2/3 compatible comment-based type hints that would help porting code from 2 to 2+3 to 3. While migrating their code from Python 2 to 3 users are most likely to discover the following types of text/binary errors (presumably, in the descending order of their frequency in typical code): 1. Implicit text/binary conversions removed in Python 3 2. Calling changed APIs that accept or return text/binary data 3. Calling removed/changed methods of text/binary types 4. Overriding special text/binary methods and using the related built-ins (`str()`, `repr()`, `unicode()`) Only the first two types of errors -- implicit conversions and calling changed text/binary APIs -- depend on being able to express the semantics of Python 2+3 compatible text/binary interfaces using type hints. PEP 0484 doesn't contain any recommendations on how to document various typical cases in text/binary APIs in order to make type hints 2+3 compatible. # Proposal This document is based on some text/binary handling options and the problems associated with them propsed at python/mypy#1141 by Jukka Lehtosalo, Guido van Rossum, and others [1]. It also takes into account the experience of the PyCharm team with their pre-PEP484 notation for type hints [2] and handling Python 2/3 issues reported by users in PyCharm code inspections. ## Handling removed implicit conversions In addition to the existing types (`bytes`, `str`, `unicode`, `typing.AnyStr`) let's introduce a new type for *2+3 compatible text data* -- `typing.Text` (should we add a fake built-in `unicode` type for Python 3 to type checkers instead of introducing a new name?): * `typing.Text`: text data * Python 2: `unicode` * Python 3: `str` Just to remind the semantics of the existing types: * `bytes`: binary data * Python 2: `bytes` (== `str`) * Python 3: `bytes` * `str`: "native" string, `type('foo')` * Python 2: `str` * Python 3: `str` * `unicode`: Python 2-only text data * Python 2: `unicode` * Python 3: error * `typing.AnyStr`: type variable constrained to both text and binary data With the addition of `typing.Text` it is possible to express the type analogous to `typing.AnyStr` that doesn't impose any type constraints (should we call it `typing.BaseString`?): * `typing.Union[typing.Text, bytes]`: both text and binary data when a type varibale isn't needed Using only `typing.Text`, `bytes`, `str`, and `typing.AnyStr` in the type hints for an API would mean that this API is Python 2 and 3 compatible in respect to implicit text/binary conversions. For Python 2 we should *not* have the implicit `str` -> `unicode` promotion since it hides errors related to implicit conversions. For 7-bit ASCII string literals in Python 2 type checkers should infer special internal types `typing._AsciiStr` and `typing._AsciiUnicode` that are compatible with both `str` and `unicode` (a *special type-checking rule* is needed): class _AsciiStr(str): pass class _AsciiUnicode(unicode): pass The details of inferring ASCII types are up to specific type checkers. In the 2+3 mode type checkers should show errors when comment- or stub- based type hints contain `unicode`. ## Examples of typical 2+3 functions A function that accepts "native" strings. It uses implicit ASCII unicode-to-str conversion at runtime in Python 2 and accepts only text data in Python 3: def getattr(o: Any, name: str, default: Any = None) -> Any: ... A function that does implicit str-to-unicode conversion at runtime in Python 2 and accepts only text data in Python 3: def hello_rus(name: Text) -> Text: return u'Привет, ' + name A function that transforms text-to-text or binary-to-binary or handles both text and binary data in some other way in both Python 2 and 3: def listdir(path: AnyStr) -> AnyStr: ... A function that works with both text and binary data in Python 2 and 3, where the author of the function some reason doens't want to have a type variable associated with `AnyStr`: def upper_len(s: Union[bytes, Text]) -> int: return len(s.upper()) A PEP-3333 compatible WSGI app function that uses "native" strings for environ and headers data while returning an iterable over binary data in both Python 2 and 3: def app(environ: Dict[str, Any], start_response: Callable[[str, List[Tuple[str, str]]], None]) \ -> Iterable[bytes]: ... A type inference example that features a type checker being able to infer `typing._AsciiStr` or `typing._AsciiUnicode` types for Python 2 using the functions defined above: method_name = u'update' # _AsciiUnicode getattr({}, method_name) # OK, implicit ASCII-only unicode-to-bytes in Py2 nonascii_data = b'\xff' # _AsciiStr hello_rus(nonascii_data) # Type checker warning # Non-ASCII bytes are not compatible with Text # _AsciiUnicode + bytes u'foo' + b'\xff' # Type checker warning # Non-ASCII bytes are not compatible with Text def f(x: AnyStr, y: AnyStr) -> AnyStr: return os.path.join('base', x, y) # _AsciiStr compatible with AnyStr # since it's compatible with # both str and unicode There are cases mentioned in [1] where more advanced type inference rules are required in order to be able to handle ASCII types. It remains unclear if these rules would be easy enough to implement in type checkers. ## Handling other types of text/binary errors No new types besides `typing.Text` are needed in order to find errors of the other types of errors listed in the Rationale section. Based on the type hints that use the above text / binary types, type checkers in the 2+3 mode should show errors when the user accesses the attributes of these types not available in both Python 2 and Python 3. [1]: https://github.com/python/mypy/issues/1141 [2]: https://github.com/JetBrains/python-skeletons#types [3]: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-a… -- Andrey Vlasovskikh Web: http://pirx.ru/ [View Less]

6 20

Add citation() to site.py
by Steven D'Aprano March 24, 2016

March 24, 2016

The standard site.py module adds pseudo-builtins: copyright credits exit help license quit I suggest one more: citation(). Python is being used heavily in scientific and academic fields, where it is often the convention to provide citations and references to the software used. The question of how to cite Python comes up from time to time, e.g.: https://mail.python.org/pipermail/tutor/2016-March/108460.html http://academia.stackexchange.com/questions/5482/how-do-… [View More]i-reference-the-pyt… http://www.gossamer-threads.com/lists/python/python/105846 http://grokbase.com/t/python/python-list/04684fggwd/citing-python I think that having a standard answer available for this question is good practice, and will help strength Python's position as a good scientific language. SciPy, SymPy and iPython all document how they prefer to be cited: https://scipy.org/citing.html https://github.com/sympy/sympy#citation http://ipython.org/citing.html As do other languages such as Mathematica: http://support.wolfram.com/kb/472 But I think it would be a good idea to emulate the R language, which provides a function that gives the prefered citation. At the R prompt: > citation() To cite R in publications use: R Core Team (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/. A BibTeX entry for LaTeX users is @Manual{, title = {R: A Language and Environment for Statistical Computing}, author = {{R Core Team}}, organization = {R Foundation for Statistical Computing}, address = {Vienna, Austria}, year = {2014}, url = {http://www.R-project.org/}, } We have invested a lot of time and effort in creating R, please cite it when using it for data analysis. See also ‘citation("pkgname")’ for citing R packages. * * * If you provide a package name such as "splines", the R's output is the same except that the first line starts with: "The ‘splines’ package is part of R." (I don't have any third-party packages installed to test, but presumably they will offer customised output.) My suggestion is that we follow R's lead and add a citation() function to site.py which gives a suggested, standard, citation string that people can copy. Obviously we cannot expect to match all standard formats, but we can provide one and folks can convert to the format their university of journal requires. (Converting between academic reference styles is out of scope of this proposal.) The initial implementation might look something like this: year = platform.python_build()[1].split()[2] vers = platform.python_version() _CITE = """\ To cite Python in publications, please use: Python Core Team ({}). Python {}: A dynamic, open source programming language. Python Software Foundation. URL https://www.python.org/. """.format(year, vers) def citation(): print(_CITE) Or perhaps it should use the same implementation as copyright, credits and license. Thoughts? -- Steve [View Less]

16 39

Fwd: [Python Ideas] Random weighted choice idea to add to random lib
by Steven Basart March 23, 2016

March 23, 2016

Hello I'm going a change from my example code and talk about the suggestions I would make to the issue I mentioned because as you said it does have accumulation of floating point errors and is a bit simplistic. Maybe it is better to be simple but not at the expense of gross inefficiency of (case of O(log(n)) vs O(n)). So here's how I'd change his code from the issue here: http://bugs.python.org/issue18844 the code on here: http://bugs.python.org/review/18844/patch/12674/46425 I've marked … [View More]what I would insert with a plus and a remove with a minus at the beginning of the line First only expose the weighted_choice function and leave the generator as the backend. Keep it simple. so from here on I'd change weighted_choice_generator to _weighted_choice_generator - "SystemRandom", "weighted_choice_generator", "weighted_choice"] + "SystemRandom", "weighted_choice"] Second we could store the previous data to prevent having to recreate the binary tree def weighted_choice(self, data): + # we could store a copy of the data to prevent remaking the tree however + # this will take up order(n) space to go to sublinear time in search + # or store a hash of the value + if self.hashval == hash(tuple(data)): + return next(self._weighted_choice_generator(data)) + else: + return self._weighted_choice_generator(data) + # Other approach is to do away with the function above and simply rename the function + # below as weighted_choice because even this would remake the tree but efficiently find the value each time def _weighted_choice_generator(self, data): I dislike of having to create two new functions into random lib API when I think only really one is required. Ultimately I think we could do away with the top function and simply keep the generator function but call it weighted_choice. On Wed, Mar 23, 2016 at 12:11 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: > On Mar 23, 2016, at 09:43, Steven Basart <xksteven(a)uchicago.edu> wrote: >> >> Hello Andrew, >> >>> On Wed, Mar 23, 2016 at 11:13 AM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >>> >>> On Mar 23, 2016, at 07:51, Steven Basart <xksteven(a)uchicago.edu> wrote: >>> >>> Hello, >>> >>> I had an idea that has in some form been proposed before. I think a random weighted choice function would be a very good addition to the random module in the python standard library. >>> >>> Here's a basic implementation of what's required albeit slightly inefficient as the search is linear as opposed to creating a binary tree and taking log n time to find the correct choice. pmf is short for probability mass function. It doesn't handle all error cases but it is more of an example to show the idea. >>> >>> >>> def randweighted(self,pmf=list): >>> """Return an int in the range [0,len(list)] weighted by list of values. >>> Raises error if list is empty. >>> """ >>> if(not pmf): >>> raise ValueError("pmf is equal to empty list") >>> cummassf = [sum(pmf[:i]) for i in pmf] >>> randVal = self._randbelow(len(pmf)) >>> for x in cummassf: >>> if randVal < x: >>> return x >>> >>> >>> Why a default value of list, especially since that's just going to raise a TypeError later on? >>> >>> Why the (unpythonic) test for "pmf == []" instead of just testing "if not pmf:"? Then it'll work on, say, tuples or third-party sorted lists or any other sequence. (With a tiny change, you can make it work on any iterable, but I don't know whether that's desirable.) >>> >>> The x[:i] is accessing an unbound local named x. What's it supposed to be? >> It was supposed to be >> >> cummassf = [sum(pmf[:i]) for i in pmf] >> >> Sorry about that I updated the above example accordingly. >> >>> Whatever x is, each element of cummassf is going to be some kind of sequence, so randVal < x on each element is going to raise a TypeError. Maybe you wanted a sum somewhere? >> Yes this is correct and I updated how it was supposed to be above. >>> If you fix that, doesn't this accumulate floating point errors, so there's a significant chance the result won't add up to 1.0 even if all of the weights are correct to 1 ULP? >>> >>> If the weights _aren't_ correct, surely you'd want a ValueError or something. This code will just return None if the sum is too low, and silently choose from the first 1.0 if it's too high. (I think np.random.choice raises ValueError if not isclose(sum(p), 1.0), but I'd have to check.) >> I think >> To fix this we could we could do one of two options: >> >> 1) Normalize the inputs >> total = 0 >> total += sum(pmf) > > Why not just total = sum(pmf)? > > Also, this isn't normalizing. Normalizing means scaling by the appropriate factor. > >> check if total is close to 1.0 >> then >> ... (same as code above) >> >> finally set cummassf[-1] = 1.0 >> >> 2) Scale the random variable by the total amount. In this case there >> is no need to take into account the floating point error. > > There very definitely is a need to take into account the floating point error! Adding an extra operation (a division) doesn't make the accumulated error go away, it makes it worse. > > I believe PEP 450 (for the statistics module) has a good discussion of accumulating errors in running sums, and how to deal with it. > > Meanwhile, from a very quick survey, it looks like many implementations allow integer weights (which you get directly out of Counter) and do exact rather than inexact math in those cases. That seems like a good idea. > >> 3) The last idea I had was to instead of returning the value we could >> return the index of which position to get as well as the actual value. >> index = 0 >> for x in cummassf: >> index += 1 >> if randVal < x: >> return (index, pmf[index]) > > Use enumerate; it's a builtin for a reason. > > Meanwhile, what use is returning the weight at all? What would a caller do with it? > > But again, why are you trying to write this from scratch if someone already wrote a working version? > >>> Finally, this is clearly a weighted randrange (with a fixed min of 0), but it claims to be a weighted choice. Obviously you can build the latter out of the former (e.g., take a val:weight mapping, call randweighted on the values, then do next(islice(keys, result, None)), but you do need to actually build it (and show what the API for it will be) if that's the motivating reason. (Also, I think an _unnormalized_ val:weight mapping might be a better API--after all, that means you can immediately call it on a Counter--but you probably want to look at what numpy, R, Mathematica, etc. do...) >>> >>> There was a ticket that got started before. It has since stalled because I believe the original author intended the library to get called over an over so they came up with generators which would be more efficient if passing in the same list. Otherwise there isn't really a point to using generators in my opinion. >>> >>> http://bugs.python.org/issue18844 >>> >>> >>> Why not just pick up from their original implementation, fix any bugs, and propose that, instead of starting over from scratch? >> I've never contributed to the python package before so I'm honestly >> not sure how to go about updating the previous ticket. I just knew >> how to find it. >> I did check out the source code from mercurial and started asking on >> irc about it. Someone there suggested I present my idea here. >> I wanted to give a simple version of what the code would perhaps look >> like in the python-ideas mailing list and also discuss the main api of >> what the function would do. I just believe it is a really useful >> function and having it available for many users would save time. > > Sure, but you don't seem to be presenting a simplified version of the code you found, but instead writing completely different code that doesn't work, doesn't have the API you want, and doesn't make it clear what you're trying to do. Again, why not just post working code with the right interface by copying the existing implementation or stripping it down? > > Meanwhile, there definitely is a point to the generators. It's very common to want to do N random choices in a row. For unweighted choice, that's fine; just call choice N times. But for weighted choice, there's a linear-time setup followed by a sublinear-time choice function, so just calling the whole thing N times will take O(NM) time, while doing the setup only once takes O(M+NlogM) or something. Most of the functions from NumPy and other languages that I found in a quick google do this by returning N values instead of 1, which works, but it seems like a generator is a much more pythonic way of implementing the same idea. > > Also, it doesn't seem to meaningfully complicate things. With your implementation (once you get it working), all you have to do is wrap the last bit in a "while True:" and change the "return" to "yield" and you're done. > > [View Less]

1 0

[Python Ideas] Random weighted choice idea to add to random lib
by Steven Basart March 23, 2016

March 23, 2016

Hello, I had an idea that has in some form been proposed before. I think a random weighted choice function would be a very good addition to the random module in the python standard library. Here's a basic implementation of what's required albeit slightly inefficient as the search is linear as opposed to creating a binary tree and taking log n time to find the correct choice. pmf is short for probability mass function. It doesn't handle all error cases but it is more of an example to show the … [View More]

3 2

Re: [Python-ideas] Integrate some itertools into the Python syntax
by Franklin? Lee March 23, 2016

March 23, 2016

The following syntax is not a real proposal, because there are way too many places it intrudes on, and you wouldn't be able to define a generator which is also a callable. That said... Slicing a list is fundamentally different from slicing an iterable. One changes state, necessarily. Every time you say "foo[:2]", it will give a different result, until you get no result at all. Instead, I'll write it "foo(:2)", and ignore the fact that that's what functions look like, and whether we CAN extend … [View More]function syntax to make this legal. Using parens reflects the syntax difference between list comprehensions and generator expressions. On Mar 21, 2016 7:36 PM, "Chris Angelico" <rosuav(a)gmail.com> wrote: > > On Tue, Mar 22, 2016 at 10:06 AM, Michel Desmoulin > <desmoulinmichel(a)gmail.com> wrote: > > > > Make slicing accept callables > > ============================= > > > > So my first proposal is to be able to do: > > > > def stop(element): > > return element > 4 > > print(numbers[:stop]) > > > > It's quite pythonic, easy to understand : the end of the slice is when > > this condition is met. Any not the strange way takewhile work, which is > > "carry on as long as this condition is met". > > > > We could also extend itertools.islice to accept such parameter. I like this. I don't think it should happen, due to existing parts of the language it might step on, but I like it. foo(f:g:h) will mean filter(h, takewhile(g, dropwhile(negate(f), foo))) (possibly clearer if we imagine being able to write: Itertooler(foo).dropwhile(negate(f)).takewhile(g).filter(h) ) where def negate(f): def negate_f(*args, **kwargs): return not f(*args, **kwargs) return negate_f > > Slicing any iterable > > ====================== > > > > So the second proposal is to allow: > > > > def func_accepting_any_iterable(foo): > > return bar(foo[3:7]) > > > > The slicing would then return a list if it's a list, a typle if it's a > > tuple, and a islice(generator) if it's a generator. If somebody uses a > > negative index, it would then raises a ValueError like islice. > > > > This would make duck typing and iteration even easier in Python. > > Again, while I am sympathetic to the problem, it's actually very hard; > islice always returns the same kind of thing, but slicing syntax can > return all manner of different things, because it's up to the object > on the left: What if `foo(3:5)` always returned a lazy iterator? (Then `iter(o)` == `o(:)`?) [View Less]

2 1

Unified database exception structure
by Sjoerd Job Postmus March 19, 2016

March 19, 2016

Dear all, I'm not sure if this has been discussed before, but I have a bit of a concern with respect to the exceptions defined in the DB-API 2.0 (PEP 0249), especially with respect to code written to support multiple database backends (for instance, through an ORM, or manually). >From what I understand, the PEP requires the module adhering to the database API to specify its own exceptions with pre-determined names, for instance: >>> psycopg2.DatabaseError.__mro__ (<… [View More]class 'psycopg2.DatabaseError'>, <class 'psycopg2.Error'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>) >>> sqlite3.DatabaseError.__mro__ (<class 'sqlite3.DatabaseError'>, <class 'sqlite3.Error'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>) Now, there is not a single base class (except for `Exception`) which I should feel comfortable about catching. I think it would have been better to see something like >>> psycopg2.DatabaseError.__mro__ (<class 'dbapi.DatabaseError'>, <class 'dbapi.Error'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>) >>> sqlite3.DatabaseError.__mro__ (<class 'dbapi.DatabaseError'>, <class 'dbapi.Error'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>) Of course, this is too late to change now, and maybe I'm alone in thinking this would make more sense, and think the best we can have is to have `sqlite3.DatabaseError` (and friends) be formed by extending both `dbapi.DatabaseError` and `sqlite3.Error` (which in turn would extend `dbapi.Error`). To make that happen, I think it would be good to see one of the following situations happen: - A module cq package gets created holding the generic exception classes, and slowly (but surely) all the independent database libraries start to use those classes in their __mro__ as well. The sqlite3 library could be the first or the last, because it is in the stdlib, depending on whether the new module/package also gets to be in the stdlib. - We wait for Python 3.6 or higher to allow using virtual base classes in `except ...:` statements (Python 2.7 already allows it). This way we could create a package which just registers the relevant database libraries exception tree with its own. The downside to going with the first route is that I envision it a slow route, which means it will also be a long time before I can use it in code which uses libraries which have not been updated yet. The upside with the second route is that there is already work being done on supporting `except VirtualBaseClass:` (see https://bugs.python.org/issue12029). Also, it means that it should be possible to just write 1 package to rule them all. Another advantage is that that solution would be backwards compatible with Python 2.7. The downside is that it's not compatible with Python 3.5. Do you have any thoughts about this? Kind regards, Sjoerd Job [View Less]

2 1