From andrey.vlasovskikh at gmail.com Tue Nov 5 13:30:00 2013 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Tue, 5 Nov 2013 16:30:00 +0400 Subject: [code-quality] Python skeletons: type hinting for third-party libraries Message-ID: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> The PyCharm team would like to propose a way for specifying and sharing API definitions of third-party libraries. As there is not enough information in API definition code of Python libraries, developers of static analysis tools collect extended API data themselves and store it in their own formats. The absence of a common extended API information format makes it hard for developers and users of tools to collect and share data. Our proposal is to create a common database of static API definitions with type hints as a collection of Python files called skeletons. Static analysis tools already understand Python code, so it should be easy to start extracting API definitions from these Python skeleton files. Regular function and class definitions can be extended with additional information, e.g. for providing types of function parameters and return values. Please read the details in the python-skeletons repository on GitHub: https://github.com/JetBrains/python-skeletons Your comments and suggestions are welcome. We would like to collaborate on this initiative. -- Andrey Vlasovskikh Senior Software Developer JetBrains http://www.jetbrains.com/ "Develop with pleasure!" From carl.crowder at gmail.com Tue Nov 5 15:15:44 2013 From: carl.crowder at gmail.com (Carl Crowder) Date: Tue, 5 Nov 2013 15:15:44 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> Message-ID: <409CB926-930A-46BF-9928-3C8C2B132961@gmail.com> Hi Andrey, I definitely support some centralized collection of API data, I think it would be very useful. How do you plan to support additional libraries? I have been writing a plyint/astroid plugin for Django for example, which has (very early beginnings of) the type information you want to collect[0]. Would you expect this to be merged into python-skeletons? If so, the one single repository could become quite large and unwieldly. Separate repositories also allows faster iteration by the maintainers of the smaller subsets. Carl [0] see https://github.com/landscapeio/pylint-django/tree/develop/pylint_django/transforms/transforms , please excuse the lack of comments right now :-) On 05.11.2013, at 13:30, Andrey Vlasovskikh wrote: > The PyCharm team would like to propose a way for specifying and sharing API definitions of third-party libraries. > > As there is not enough information in API definition code of Python libraries, developers of static analysis tools collect extended API data themselves and store it in their own formats. The absence of a common extended API information format makes it hard for developers and users of tools to collect and share data. > > Our proposal is to create a common database of static API definitions with type hints as a collection of Python files called skeletons. Static analysis tools already understand Python code, so it should be easy to start extracting API definitions from these Python skeleton files. Regular function and class definitions can be extended with additional information, e.g. for providing types of function parameters and return values. Please read the details in the python-skeletons repository on GitHub: https://github.com/JetBrains/python-skeletons > > Your comments and suggestions are welcome. We would like to collaborate on this initiative. > > -- > Andrey Vlasovskikh > Senior Software Developer > JetBrains > http://www.jetbrains.com/ > "Develop with pleasure!" > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality From jeff.janies at redjack.com Tue Nov 5 17:28:36 2013 From: jeff.janies at redjack.com (Jeff Janies) Date: Tue, 5 Nov 2013 11:28:36 -0500 Subject: [code-quality] Possible Issue Message-ID: I have recently started using Flake8 for syntax checking and have found that the style checking fails to check that `if __name__ == ?__main__?:` after a class or function has the correct number of newlines preceding it. Shouldn?t this follow the same style rules as functions outside of a class? Jeff Janies From graffatcolmingov at gmail.com Wed Nov 6 01:07:18 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Tue, 5 Nov 2013 18:07:18 -0600 Subject: [code-quality] Possible Issue In-Reply-To: References: Message-ID: On Tue, Nov 5, 2013 at 10:28 AM, Jeff Janies wrote: > I have recently started using Flake8 for syntax checking and have found that the style checking fails to check that `if __name__ == ?__main__?:` after a class or function has the correct number of newlines preceding it. Shouldn?t this follow the same style rules as functions outside of a class? Hey Jeff! That may be an issue, I can not remember if the PEP specifies it or not off the top of my head, but let me give you some context (which you may already have): Flake8 wraps around 3 other tools: mccabe, pep8, and pyflakes. The behaviour you're seeing is coming from the pep8 tool. If PEP-008 (the document) specifies that this should be the case and using flake8 without any configuration does not warn you, then you should raise an issue on https://github.com/jcrocholl/pep8 Thanks for bringing this to our attention! -- Ian From ben+python at benfinney.id.au Wed Nov 6 02:58:07 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 06 Nov 2013 12:58:07 +1100 Subject: [code-quality] Possible Issue References: Message-ID: <7wli12xpe8.fsf@benfinney.id.au> Jeff Janies writes: > I have recently started using Flake8 for syntax checking and have > found that the style checking fails to check that `if __name__ == > ?__main__?:` after a class or function has the correct number of > newlines preceding it. Take care, when posting email, to use an editor that won't munge the text you're writing. Your message says ?if __name__ == ?__main__?:?, which is a SyntaxError. Easy enough to tell in this instance what you mean, but you should take care not to require *any* guessing on the part of the reader what exact code you are showing us. If you're using a tool that turns ?"? into a different character, you need to avoid that tool when composing any messages about program code. -- \ ?We now have access to so much information that we can find | `\ support for any prejudice or opinion.? ?David Suzuki, 2008-06-27 | _o__) | Ben Finney From andrey.vlasovskikh at gmail.com Wed Nov 6 13:48:05 2013 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Wed, 6 Nov 2013 16:48:05 +0400 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <409CB926-930A-46BF-9928-3C8C2B132961@gmail.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <409CB926-930A-46BF-9928-3C8C2B132961@gmail.com> Message-ID: <673B5FFF-14A6-4CB7-8DF7-D2E0CAEFD02E@gmail.com> Hi Carl, Yes, we would like to put all skeletons for third-party libraries in a single repo and manage the repo via GitHub pull requests. A single repo with works well for DefinitelyTyped, a collection of TypeScript stubs similar to Python skeletons https://github.com/borisyankov/DefinitelyTyped and other projects like Homebrew package manager for OS X that contains thousands of package entries for third-party libraries https://github.com/mxcl/homebrew Please refer to the README.md file at https://github.com/JetBrains/python-skeletons -- Andrey Vlasovskikh Senior Software Developer JetBrains http://www.jetbrains.com/ "Develop with pleasure!" From vladimir at keleshev.com Wed Nov 6 15:08:31 2013 From: vladimir at keleshev.com (Vladimir Keleshev) Date: Wed, 06 Nov 2013 15:08:31 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> Message-ID: <197061383746911@web13m.yandex.ru> Hi Andrey, I think this is a great proposition that can help many static analysis tools. However, it makes me very sad that you have chosen the docutils format for parameters and return types: :type x: float :param x: The X coordinate. :type y: float :param y: The Y coordinate. :rtype: int You should take a look at NumPy docstring conventions which are fully supported by Sphinx, and are actually human-readable. https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt Here is an example: Parameters ---------- x : float The X coordinate. y : float The Y coordinate. Returns ------- int Description of anonymous integer return value. It also allows documenting object/class attributes, and exceptions. ?Vladimir 05.11.2013, 13:30, "Andrey Vlasovskikh" : > The PyCharm team would like to propose a way for specifying and sharing API definitions of third-party libraries. > > As there is not enough information in API definition code of Python libraries, developers of static analysis tools collect extended API data themselves and store it in their own formats. The absence of a common extended API information format makes it hard for developers and users of tools to collect and share data. > > Our proposal is to create a common database of static API definitions with type hints as a collection of Python files called skeletons. Static analysis tools already understand Python code, so it should be easy to start extracting API definitions from these Python skeleton files. Regular function and class definitions can be extended with additional information, e.g. for providing types of function parameters and return values. Please read the details in the python-skeletons repository on GitHub: https://github.com/JetBrains/python-skeletons > > Your comments and suggestions are welcome. We would like to collaborate on this initiative. > > -- > Andrey Vlasovskikh > Senior Software Developer > JetBrains > http://www.jetbrains.com/ > "Develop with pleasure!" > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality From andrey.vlasovskikh at gmail.com Thu Nov 7 13:53:33 2013 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Thu, 7 Nov 2013 16:53:33 +0400 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <197061383746911@web13m.yandex.ru> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> Message-ID: <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> Hi Vladimir, > You should take a look at NumPy docstring conventions which are fully supported by Sphinx, and are actually human-readable. There are two questions here: 1. Where to put type information? * Docstrings * ReStructured Text (http://sphinx-doc.org/domains.html#the-python-domain) * Epydoc * NumPy * Expressions * Python 3 function annotations * Decorators 2. What type system to use? (its syntax and semantics, see the related work section in the Python skeletons proposal [1]) We think that the best place for type information would be function annotations, but they are available only in Python 3 and there is no de facto standard for describing types as Python expressions. We use docstring-based types in Python skeletons only as a simple temporary solution, since we believe that the effort should be put in creating an expression-based type system for Python that would be acceptable for a large number of static analysis projects and regular Python developers. We would like to discuss the current implementations [1] and come up with a unified expression-based language (perhaps, supported by a library) for describing Python types, or at least, a set of common patterns (e.g. using a class value as a type of its instances). [1]: https://github.com/JetBrains/python-skeletons#related-work -- Andrey Vlasovskikh Senior Software Developer JetBrains http://www.jetbrains.com/ "Develop with pleasure!" From vladimir at keleshev.com Thu Nov 7 15:14:41 2013 From: vladimir at keleshev.com (Vladimir Keleshev) Date: Thu, 07 Nov 2013 15:14:41 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> Message-ID: <311511383833681@web14j.yandex.ru> Hi Andrey, 1. I'm a docstring bigot, so the only thing I can recommend is that *if* you decide to use docstrings for type annotations, please, select the format that is readable and well-supported (i.e. NumPy). 2. I don't have much to comment on how the types should be specified, but you should take into account that python is duck-typed language. Sometimes the type is "anything that implements .read() and .write() methods", or "anything that follows `__contains__` protocol". ?Vladimir 07.11.2013, 13:53, "Andrey Vlasovskikh" : > Hi Vladimir, > >> ?You should take a look at NumPy docstring conventions which are fully supported by Sphinx, and are actually human-readable. > > There are two questions here: > > 1. Where to put type information? > > ????* Docstrings > ????????* ReStructured Text (http://sphinx-doc.org/domains.html#the-python-domain) > ????????* Epydoc > ????????* NumPy > ????* Expressions > ????????* Python 3 function annotations > ????????* Decorators > > 2. What type system to use? (its syntax and semantics, see the related work section in the Python skeletons proposal [1]) > > We think that the best place for type information would be function annotations, but they are available only in Python 3 and there is no de facto standard for describing types as Python expressions. > > We use docstring-based types in Python skeletons only as a simple temporary solution, since we believe that the effort should be put in creating an expression-based type system for Python that would be acceptable for a large number of static analysis projects and regular Python developers. > > We would like to discuss the current implementations [1] and come up with a unified expression-based language (perhaps, supported by a library) for describing Python types, or at least, a set of common patterns (e.g. using a class value as a type of its instances). > > [1]: https://github.com/JetBrains/python-skeletons#related-work > > -- > Andrey Vlasovskikh > Senior Software Developer > JetBrains > http://www.jetbrains.com/ > "Develop with pleasure!" From noisecapella at gmail.com Thu Nov 7 19:22:39 2013 From: noisecapella at gmail.com (George Schneeloch) Date: Thu, 7 Nov 2013 13:22:39 -0500 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <311511383833681@web14j.yandex.ru> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <311511383833681@web14j.yandex.ru> Message-ID: As long as the docstrings can be easily parsed (I think there's I library, haven't tried it myself) then would it matter much? It seems like a large change for a small benefit -George On Nov 7, 2013 9:17 AM, "Vladimir Keleshev" wrote: > Hi Andrey, > > 1. I'm a docstring bigot, so the only thing I can recommend is that *if* > you decide to use docstrings for type annotations, please, select the > format that is readable and well-supported (i.e. NumPy). > > 2. I don't have much to comment on how the types should be specified, but > you should take into account that python is duck-typed language. Sometimes > the type is "anything that implements .read() and .write() methods", or > "anything that follows `__contains__` protocol". > > ?Vladimir > > > 07.11.2013, 13:53, "Andrey Vlasovskikh" : > > Hi Vladimir, > > > >> You should take a look at NumPy docstring conventions which are fully > supported by Sphinx, and are actually human-readable. > > > > There are two questions here: > > > > 1. Where to put type information? > > > > * Docstrings > > * ReStructured Text ( > http://sphinx-doc.org/domains.html#the-python-domain) > > * Epydoc > > * NumPy > > * Expressions > > * Python 3 function annotations > > * Decorators > > > > 2. What type system to use? (its syntax and semantics, see the related > work section in the Python skeletons proposal [1]) > > > > We think that the best place for type information would be function > annotations, but they are available only in Python 3 and there is no de > facto standard for describing types as Python expressions. > > > > We use docstring-based types in Python skeletons only as a simple > temporary solution, since we believe that the effort should be put in > creating an expression-based type system for Python that would be > acceptable for a large number of static analysis projects and regular > Python developers. > > > > We would like to discuss the current implementations [1] and come up > with a unified expression-based language (perhaps, supported by a library) > for describing Python types, or at least, a set of common patterns (e.g. > using a class value as a type of its instances). > > > > [1]: https://github.com/JetBrains/python-skeletons#related-work > > > > -- > > Andrey Vlasovskikh > > Senior Software Developer > > JetBrains > > http://www.jetbrains.com/ > > "Develop with pleasure!" > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir at keleshev.com Thu Nov 7 19:33:49 2013 From: vladimir at keleshev.com (Vladimir Keleshev) Date: Thu, 07 Nov 2013 19:33:49 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <311511383833681@web14j.yandex.ru> Message-ID: <131051383849229@web18g.yandex.ru> An HTML attachment was scrubbed... URL: From vladimir at keleshev.com Fri Nov 8 11:08:39 2013 From: vladimir at keleshev.com (Vladimir Keleshev) Date: Fri, 08 Nov 2013 11:08:39 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <527CB6B4.7010604@redhat.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <527CB6B4.7010604@redhat.com> Message-ID: <191531383905319@web27m.yandex.ru> > You could also put asserts (and expressions which assert the presence of > certain attributes) in function bodies. Nice idea?that seems to give more flexibility, but (probably) makes it harder to implement. I would love to write: assert hasattr(file_like, 'read') and hasattr(file_like, 'write') Or maybe: assert file_like.read and file_like.write On the other hand, documenting type information in docstrings is also nice. ?Vladimir 08.11.2013, 11:02, "Florian Weimer" : > On 11/07/2013 01:53 PM, Andrey Vlasovskikh wrote: > >>> ?You should take a look at NumPy docstring conventions which are fully supported by Sphinx, and are actually human-readable. >> ?There are two questions here: >> >> ?1. Where to put type information? >> >> ??????* Docstrings >> ??????????* ReStructured Text (http://sphinx-doc.org/domains.html#the-python-domain) >> ??????????* Epydoc >> ??????????* NumPy >> ??????* Expressions >> ??????????* Python 3 function annotations >> ??????????* Decorators > > You could also put asserts (and expressions which assert the presence of > certain attributes) in function bodies. ?Existing analysis tools should > already process those and would only need to implement different error > reporting in case a mismatch is detected (because the error is strictly > with the caller, not in the skeleton module). > > -- > Florian Weimer / Red Hat Product Security Team From vladimir at keleshev.com Fri Nov 8 11:19:16 2013 From: vladimir at keleshev.com (Vladimir Keleshev) Date: Fri, 08 Nov 2013 11:19:16 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <527CB8FE.60205@redhat.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <527CB6B4.7010604@redhat.com> <191531383905319@web27m.yandex.ru> <527CB8FE.60205@redhat.com> Message-ID: <167101383905956@web26j.yandex.ru> I agree that not writing any type annotations is the best, but that would require complex type inference. OCaml does type inference for it's duck-typed objective extension, but I'm not sure that such type inference is possible for a dynamic language like Python. 08.11.2013, 11:12, "Florian Weimer" : > On 11/08/2013 11:08 AM, Vladimir Keleshev wrote: > >>> ?You could also put asserts (and expressions which assert the presence of >>> ?certain attributes) in function bodies. >> ?Nice idea?that seems to give more flexibility, but (probably) makes it harder to implement. >> >> ?I would love to write: >> >> ??????assert hasattr(file_like, 'read') and hasattr(file_like, 'write') >> >> ?Or maybe: >> >> ??????assert file_like.read and file_like.write > > I actually expect something like this: > > ?????file_like.read(0) > ?????file_like.write("") > > No assert statements, just straight-line code that is supposed to be > easy to analyze. ?If the analyzer cannot use the information that > file_like must have a callable read attribute that takes an integer > argument, it could still use the information that a "read" attribute > must be present. > > -- > Florian Weimer / Red Hat Product Security Team From fweimer at redhat.com Fri Nov 8 11:02:28 2013 From: fweimer at redhat.com (Florian Weimer) Date: Fri, 08 Nov 2013 11:02:28 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> Message-ID: <527CB6B4.7010604@redhat.com> On 11/07/2013 01:53 PM, Andrey Vlasovskikh wrote: >> You should take a look at NumPy docstring conventions which are fully supported by Sphinx, and are actually human-readable. > > There are two questions here: > > 1. Where to put type information? > > * Docstrings > * ReStructured Text (http://sphinx-doc.org/domains.html#the-python-domain) > * Epydoc > * NumPy > * Expressions > * Python 3 function annotations > * Decorators You could also put asserts (and expressions which assert the presence of certain attributes) in function bodies. Existing analysis tools should already process those and would only need to implement different error reporting in case a mismatch is detected (because the error is strictly with the caller, not in the skeleton module). -- Florian Weimer / Red Hat Product Security Team From fweimer at redhat.com Fri Nov 8 11:12:14 2013 From: fweimer at redhat.com (Florian Weimer) Date: Fri, 08 Nov 2013 11:12:14 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <191531383905319@web27m.yandex.ru> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <527CB6B4.7010604@redhat.com> <191531383905319@web27m.yandex.ru> Message-ID: <527CB8FE.60205@redhat.com> On 11/08/2013 11:08 AM, Vladimir Keleshev wrote: >> You could also put asserts (and expressions which assert the presence of >> certain attributes) in function bodies. > > Nice idea?that seems to give more flexibility, but (probably) makes it harder to implement. > > I would love to write: > > assert hasattr(file_like, 'read') and hasattr(file_like, 'write') > > Or maybe: > > assert file_like.read and file_like.write I actually expect something like this: file_like.read(0) file_like.write("") No assert statements, just straight-line code that is supposed to be easy to analyze. If the analyzer cannot use the information that file_like must have a callable read attribute that takes an integer argument, it could still use the information that a "read" attribute must be present. -- Florian Weimer / Red Hat Product Security Team From fweimer at redhat.com Fri Nov 8 12:15:11 2013 From: fweimer at redhat.com (Florian Weimer) Date: Fri, 08 Nov 2013 12:15:11 +0100 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <167101383905956@web26j.yandex.ru> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <527CB6B4.7010604@redhat.com> <191531383905319@web27m.yandex.ru> <527CB8FE.60205@redhat.com> <167101383905956@web26j.yandex.ru> Message-ID: <527CC7BF.7050302@redhat.com> On 11/08/2013 11:19 AM, Vladimir Keleshev wrote: > I agree that not writing any type annotations is the best, but that would require complex type inference. OCaml does type inference for it's duck-typed objective extension, but I'm not sure that such type inference is possible for a dynamic language like Python. The skeleton files wouldn't contain an actual Python implementation, only a rough summary from which requirements can be extracted. Especially conditional code should be kept to a bare minimum. Type inference (which would reconstruct some principal type of the function, needing a suitable type system that can express such type) is not required for the skeleton, you can use the same type propagation engine you use for extracting information in client code which you can then check against the skeleton. -- Florian Weimer / Red Hat Product Security Team From kenn.knowles at gmail.com Fri Nov 8 19:50:57 2013 From: kenn.knowles at gmail.com (Kenn Knowles) Date: Fri, 8 Nov 2013 13:50:57 -0500 Subject: [code-quality] Python skeletons: type hinting for third-party libraries Message-ID: Hello, This seems like a great project. I would *love* to see a more widespread type language for Python. I am commenting less on the skeleton collection because I think just mimicking DefinitelyTyped and Homebrew is such a good way to do it that I support it completely. I will definitely contribute skeletons for my dependencies in order to get better checking for my projects. For the type language... I am *less* concerned with: - *The exact concrete syntax for types.* I think Foo -> Bar and [Foo] and{Foo:Bar}are all nicer than Function[Foo,Bar] and List[Foo] and dict[Foo,Bar], but as long as the underlying type system has these concepts and there is *some* syntax for them (which there is not in typeannotations, mypy, or any contractl ibrary that I know of) then everything is fine. And we can always support shorthands in addition to the direct __repr__ style of writing these types. - *How the types get into Python code.* This is probably best done a few different way. The way that typeannotations and mypy do it with Python 3 annotations is lovely, but we cannot wait for Python 3 support everywhere. Docstrings in a various of ways are fine. This decision should be independent from the type language itself. - *How the types are checked*. Static checking is super, and dynamic checking is really helpful too, but as the docstring formats show it is really amazing to even just have the types for a human to read, and document your intent in a rigorous way. Wild monkey patching and reflection will probably always make it hard to be actually type safe and that would be unpythonic anyhow. I am *most* concerned with the *abstract* type language. Here are notes on the current proposal (and some context): - *The type language must have function types.* It is really disappointing when the only when I can describe a function's type is interleaved with its definition. It is very important to be able to say "method some_method has type (int, datetime, {int:str}, *str) -> str. Without this, I don't even have the words to describe the method. - *The type language should really have "static duck types"*. These were mentioned briefly by Vladimir. The technical term is row types, and they are only currently in python-rightarrow with the concrete syntax like object(self_type, field: type, field: type, ...). Without this, I don't have the words to describe an object. It is not really Pythonic to describe it in terms of its implementation when all that matters is that it has certain methods (of certain types). - *The type language should allow types for *args and **kwargs*. I do not see this in the current proposal. The current python-rightarrow contains one idea but I think it can be simplified to e.g. (Foo, *Bar, **Bizzle) -> Baz (since kwargs has to have str keys -- I think? -- you only give the type of its values.) - *Generics are just variables and should allow self-documenting names. *Just a little thing; Reserving T-Z for generics makes it so you cannot give them useful names. To be very clear: I believe these things about the whole project: - *I think the abstract type language can be describe without being too rigid about the concrete syntax.* The actual ways in which Python data structure are put together is not going to change any time soon, so if we get the abstract part right, there can be innovation in the concrete syntax, shorthands, etc. In fact, tuples, lists, dicts, and functions are all "real" things that exist pretty much independently of Python. I feel sort of the same way about row types, etc. So we may as well use the universal abstract syntax for them and give it a Pythonic concrete syntax. - *I think all of that can be defined without having a type checker (or even a type system) to handle all of them perfectly. *Any type that is too hard for a type checker can be treated as "unknown". Questions: - The type language proposal should go in its own repo perhaps? - How will license be handled for the major central repo of skeletons? I would prefer a global BSD/Apache/MIT/whatever style license and make that clear in CONTRIBUTING. Disclaimer: I am the author of python-rightarrow which is really an unimportant and barely-started side project to develop a simple type language for Python. I have a lot of experience with programming language and type checking / type inference / static-dynamic hybrid type implementation. I hope the result of this discussion makes my project obsolete. Cheers, - Kenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrey.vlasovskikh at gmail.com Mon Nov 11 13:59:43 2013 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Mon, 11 Nov 2013 16:59:43 +0400 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <311511383833681@web14j.yandex.ru> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <311511383833681@web14j.yandex.ru> Message-ID: <04EDAE9F-689B-4893-A38C-B49236C42419@gmail.com> Vladimir, > 2. I don't have much to comment on how the types should be specified, but you should take into account that python is duck-typed language. Sometimes the type is "anything that implements .read() and .write() methods", or "anything that follows `__contains__` protocol?. Yes, duck typing is one of the issues a type system for Python has to address. Possible solutions: * Add structural typing (anything with a particular set of attributes will do) * Use abstract base classes like in 'collections' and 'numbers' stdlib modules * Don't annotate duck-typed values -- Andrey Vlasovskikh Senior Software Developer JetBrains http://www.jetbrains.com/ "Develop with pleasure!" From andrey.vlasovskikh at gmail.com Mon Nov 11 14:00:30 2013 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Mon, 11 Nov 2013 17:00:30 +0400 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: <527CB8FE.60205@redhat.com> References: <9F79E0C5-A6AA-4D50-AB20-85B1EC2057CB@gmail.com> <197061383746911@web13m.yandex.ru> <60EA606B-270A-4A98-BD6C-C2AD9A1AEB27@gmail.com> <527CB6B4.7010604@redhat.com> <191531383905319@web27m.yandex.ru> <527CB8FE.60205@redhat.com> Message-ID: Florian, >>> You could also put asserts (and expressions which assert the presence of >>> certain attributes) in function bodies. >> >> Nice idea?that seems to give more flexibility, but (probably) makes it harder to implement. >> >> I would love to write: >> >> assert hasattr(file_like, 'read') and hasattr(file_like, 'write') >> >> Or maybe: >> >> assert file_like.read and file_like.write > > I actually expect something like this: > > file_like.read(0) > file_like.write("") > > No assert statements, just straight-line code that is supposed to be easy to analyze. If the analyzer cannot use the information that file_like must have a callable read attribute that takes an integer argument, it could still use the information that a "read" attribute must be present. Currently Python skeletons already include return values that are easy to parse. Static analyzers that do not support current type conventions of Python skeletons can already benefit from skeletons, because most of them can infer return types of functions. Note that sometimes it is hard to construct a simple return value of the given type. How would you annotate a function that takes a string and does something complex with it? def f(s): s.lower() would be not enough since 'f' may require a parameter with more methods than just 'lower'. Also it would be confusing from the user experience perspective. Compare these error messages from a static analysis tool: First argument of 'f' must be of type 'str' First argument of 'f' must have 'lower', 'strip' and '__getitem__' defined on it It seems that the most simple way of documenting functions is putting types of their parameters and return values. -- Andrey Vlasovskikh Senior Software Developer JetBrains http://www.jetbrains.com/ "Develop with pleasure!" From kenn.knowles at gmail.com Sat Nov 16 20:36:51 2013 From: kenn.knowles at gmail.com (Kenn Knowles) Date: Sat, 16 Nov 2013 14:36:51 -0500 Subject: [code-quality] Python skeletons: type hinting for third-party libraries In-Reply-To: References: Message-ID: Hello again, Perhaps I missed if it was already mentioned, but I thought I would ensure that PySonar was in the conversation as it was only recently released as open source. It seems to share goals with mypy and PyCharm, and has perhaps already been used by SourceGraph to do a light & broad analysis of many project on github. - https://github.com/yinwang0/pysonar2/ - https://sourcegraph.com (unfortunately lacking the type information on the site like Hoogle has, but it seems like the tool does types as well) - (photo of slide from which I make quick guesses) https://twitter.com/sqs/status/400834847855611904/photo/1 It would be nice to get all these project collaborating on the skeletons. - Kenn On Fri, Nov 8, 2013 at 1:50 PM, Kenn Knowles wrote: > Hello, > > This seems like a great project. I would *love* to see a more widespread > type language for Python. I am commenting less on the skeleton collection > because I think just mimicking DefinitelyTyped and Homebrew is such a good > way to do it that I support it completely. I will definitely contribute > skeletons for my dependencies in order to get better checking for my > projects. > > For the type language... > > I am *less* concerned with: > > - *The exact concrete syntax for types.* I think Foo -> Bar and [Foo]and{Foo:Bar}are all nicer than > Function[Foo,Bar] and List[Foo] and dict[Foo,Bar], but as long as the > underlying type system has these concepts and there is *some* syntax > for them (which there is not in typeannotations, mypy, or any contractl > ibrary that I know of) then everything is fine. And we can always support > shorthands in addition to the direct __repr__ style of writing these > types. > > > - *How the types get into Python code.* This is probably best done a > few different way. The way that typeannotations and mypy do it with Python > 3 annotations is lovely, but we cannot wait for Python 3 support > everywhere. Docstrings in a various of ways are fine. This decision should > be independent from the type language itself. > > > - *How the types are checked*. Static checking is super, and dynamic > checking is really helpful too, but as the docstring formats show it is > really amazing to even just have the types for a human to read, and > document your intent in a rigorous way. Wild monkey patching and reflection > will probably always make it hard to be actually type safe and that would > be unpythonic anyhow. > > > I am *most* concerned with the *abstract* type language. Here are notes > on the current proposal (and some context): > > - *The type language must have function types.* It is really > disappointing when the only when I can describe a function's type is > interleaved with its definition. It is very important to be able to say > "method some_method has type (int, datetime, {int:str}, *str) -> str. > Without this, I don't even have the words to describe the method. > > > - *The type language should really have "static duck types"*. These > were mentioned briefly by Vladimir. The technical term is row types, and > they are only currently in python-rightarrow with the concrete syntax like object(self_type, > field: type, field: type, ...). Without this, I don't have the words > to describe an object. It is not really Pythonic to describe it in terms of > its implementation when all that matters is that it has certain methods (of > certain types). > > > - *The type language should allow types for *args and **kwargs*. I do > not see this in the current proposal. The current python-rightarrow > contains one idea but I think it can be simplified to e.g. (Foo, *Bar, > **Bizzle) -> Baz (since kwargs has to have str keys -- I think? -- you > only give the type of its values.) > > > - *Generics are just variables and should allow self-documenting > names. *Just a little thing; Reserving T-Z for generics makes it so > you cannot give them useful names. > > > To be very clear: I believe these things about the whole project: > > - *I think the abstract type language can be describe without being > too rigid about the concrete syntax.* The actual ways in which Python > data structure are put together is not going to change any time soon, so if > we get the abstract part right, there can be innovation in the concrete > syntax, shorthands, etc. In fact, tuples, lists, dicts, and functions are > all "real" things that exist pretty much independently of Python. I feel > sort of the same way about row types, etc. So we may as well use the > universal abstract syntax for them and give it a Pythonic concrete syntax. > > > - *I think all of that can be defined without having a type checker > (or even a type system) to handle all of them perfectly. *Any type > that is too hard for a type checker can be treated as "unknown". > > > Questions: > > - The type language proposal should go in its own repo perhaps? > > > - How will license be handled for the major central repo of skeletons? > I would prefer a global BSD/Apache/MIT/whatever style license and make that > clear in CONTRIBUTING. > > > Disclaimer: I am the author of python-rightarrow which is really an > unimportant and barely-started side project to develop a simple type > language for Python. I have a lot of experience with programming language > and type checking / type inference / static-dynamic hybrid type > implementation. I hope the result of this discussion makes my project > obsolete. > > Cheers, > > - Kenn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ron.Burnett at sandisk.com Fri Nov 29 13:02:23 2013 From: Ron.Burnett at sandisk.com (Ron Burnett) Date: Fri, 29 Nov 2013 12:02:23 +0000 Subject: [code-quality] Compatibility question Message-ID: <6610AECCD68CAF4DB5A8FB337ADCD62907552DB3@SACMBXIP02.sdcorp.global.sandisk.com> Sir/Madam I am interested in determining whether asteroid 1.0.1 can be installed via the command: python setup.py install when you are using python 2.4? Regards Ron Ron Burnett Staff Test Engineer [cid:image001.png at 01CE9342.6D040E30] Links House, 15 Links Place, Edinburgh, UK, EH6 7EZ T: + 44-131-561-4852 Ron.Burnett at SanDisk.com ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 2139 bytes Desc: image001.png URL: