Thanks! There are some new changes (I fixed a race with sockets closing) and I hope to land flow control (finally) later today. Do you know what those skips are? I suspect they might be due to ssl not working for you either. :-( On Fri, Oct 18, 2013 at 10:04 AM, Richard Oudkerk <shibturn@gmail.com>wrote:

On 18/10/2013 5:31pm, Guido van Rossum wrote:

...

I'm working from home today and my Windows laptop is in the office, so I won't be able to test my latest Tulip changes on Windows (I just renamed pause to pause_reading, and hope to commit pause_reading later today). Is anyone luckier?

$ hg id 97f6f35b02e5 (asyncio) tip

$ python-release runtests.py Skipping 'test_unix_events': UNIX only ..............................**..................s...........** ......s......ss...**ssssssssssss.....s........s...**.ss..sss.....ss...** ssssssssssssss................**..............................** ..............................**..............................** ..............................**..............................** ..............................**..............................** ..............................**..............................** ..............................**..............................** ..............................**............................ ------------------------------**------------------------------**---------- Ran 544 tests in 12.790s

OK (skipped=39)

Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I

...

lost courage, so I've not tested the SSL behavior on Windows at all. Is anyone lucky enough to have a Python 3.4 alpha for Windows setup with OpenSSL already configured? If so, could you run test_asyncio for me and let me know if there are any issues? (Actually, even a working Python 3.3 setup would be useful, although you'd have to check out the Tulip repo and run the tests from there.)

$ python-release -c 'import sys; print(sys.version)' 3.4.0a3+ (default:7172135d60f6, Oct 18 2013, 17:54:18) [MSC v.1600 32 bit (Intel)]

$ python-release -m test test_asyncio [1/1] test_asyncio 1 test OK.

BTW, pcbuild.sln was not building _overlapped automatically -- I have pushed a fix.

-- Richard

-- --Guido van Rossum (python.org/~guido)

Thanks! Those are all expected (though contributions are always welcome -- not looking at you specifically :-). Does examples/fetch3.py work for you with an https URL? (Try http://dropbox.com, i.e. without 's' -- you get two redirects to https URLs. :-) On Fri, Oct 18, 2013 at 10:36 AM, Richard Oudkerk <shibturn@gmail.com>wrote:

On 18/10/2013 6:15pm, Guido van Rossum wrote:

...

Thanks! There are some new changes (I fixed a race with sockets closing) and I hope to land flow control (finally) later today.

Do you know what those skips are? I suspect they might be due to ssl not working for you either. :-(

Lack of support for subprocess, signals, ssl (with iocp), pipes, add_*() (with iocp):

$ python-release runtests.py -v 2>&1 | grep skipped test_add_signal_handler (test_events.**SelectEventLoopTests) ... skipped 'No SIGKILL' test_read_pipe (test_events.**SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_signal_handling_args (test_events.**SelectEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_**selecting (test_events.**SelectEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_**finish (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_close_client_**stream (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exec (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exitcode (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_interactive (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_kill (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_send_signal (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_shell (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr_**redirect_to_stdout (test_events.**SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_write_pipe (test_events.**SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_write_pipe_disconnect_on_**close (test_events.**SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_add_signal_handler (test_events.**ProactorEventLoopTests) ... skipped 'No SIGKILL' test_create_datagram_endpoint (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have create_datagram_endpoint()' test_create_server_ssl (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_create_ssl_connection (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_read_pipe (test_events.**ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_reader_callback (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_reader_callback_cancel (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_signal_handling_args (test_events.**ProactorEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_**selecting (test_events.**ProactorEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_**finish (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_close_client_**stream (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exec (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exitcode (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_interactive (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_kill (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_send_signal (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_shell (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr_**redirect_to_stdout (test_events.**ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_write_pipe (test_events.**ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_write_pipe_disconnect_on_**close (test_events.**ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_writer_callback (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' test_writer_callback_cancel (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' OK (skipped=39)

-- Richard

-- --Guido van Rossum (python.org/~guido)

Maybe the dummy socket returned by wrap_socket() is not acceptable for select? --Guido van Rossum (sent from Android phone) On Oct 18, 2013 11:26 AM, "Richard Oudkerk" <shibturn@gmail.com> wrote:

On 18/10/2013 6:57pm, Guido van Rossum wrote:

...

Thanks! Those are all expected (though contributions are always welcome -- not looking at you specifically :-).

Does examples/fetch3.py work for you with an https URL? (Try http://dropbox.com, i.e. without 's' -- you get two redirects to https URLs. :-)

It fails -- not sure why...

$ python-release examples/fetch3.py http://dropbox.com redirect to https://dropbox.com/ Traceback (most recent call last): File "examples/fetch3.py", line 211, in <module> main() File "examples/fetch3.py", line 206, in main body = loop.run_until_complete(fetch(**sys.argv[1], '-v' in sys.argv)) File "C:\Repos\cpython-dirty\lib\**asyncio\base_events.py", line 172, in run_until_complete self.run_forever() File "C:\Repos\cpython-dirty\lib\**asyncio\base_events.py", line 153, in run_forever self._run_once() File "C:\Repos\cpython-dirty\lib\**asyncio\base_events.py", line 576, in _run_once event_list = self._selector.select(timeout) File "C:\Repos\cpython-dirty\lib\**selectors.py", line 219, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File "C:\Repos\cpython-dirty\lib\**selectors.py", line 210, in _select r, w, x = select.select(r, w, w, timeout) OSError: [WinError 10038] An operation was attempted on something that is not a socket

-- Richard

Good sleuthing! Does the attached patch fix it? (Off-topic: the code is pretty inconsistent about catching BaseException. Maybe it shouldn't be caught at all?) On Fri, Oct 18, 2013 at 2:04 PM, Richard Oudkerk <shibturn@gmail.com> wrote:

On 18/10/2013 9:19pm, Guido van Rossum wrote:

...

Maybe the dummy socket returned by wrap_socket() is not acceptable for select?

An error

SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553)')

is being raised in _on_handshake(). This seems to result in the socket being closed without being unregistered from the selector.

select() fails before the SSLError gets reported, so it does not appear in the traceback.

-- Richard

-- --Guido van Rossum (python.org/~guido)

Thanks! That's probably fine for now -- it means the standard library doesn't know where the root certificates are. We had a huge discussion about this over on python-tulip: https://groups.google.com/forum/#!topic/python-tulip/c_lqdFjPEbE TL;DR: The stdlib openssl wrapper ought to know where each platform stores its root certificates and automatically use them, but it currently doesn't always. Users who really don't care but still want to use SSL must create an SSL context with verify_mode set to ssl.CERT_NONE (and live with the risk, obviously). This stuff passes on OS X only because there's a system openssl library that always uses the system root certificates. If anyone can help fixing the ssl.py module (or the _ssl extension) so that sslcontext.set_default_verify_paths() uses the system root certs on Windows that would be a huge help. (I have tried this on an Ubuntu box too, and there it actually works.) On Fri, Oct 18, 2013 at 3:42 PM, Richard Oudkerk <shibturn@gmail.com> wrote:

On 18/10/2013 10:37pm, Guido van Rossum wrote:

...

Good sleuthing! Does the attached patch fix it?

(Off-topic: the code is pretty inconsistent about catching BaseException. Maybe it shouldn't be caught at all?)

It fixes it in the sense of printing a sensible traceback;-)

$ PYTHONPATH='c:/Repos/tulip' /c/Repos/cpython-33/PCbuild/**python fetch3.py http://dropbox.com -v * Connecting to dropbox.com:80 using tcp * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, 199.47.217.179 * New connection ('108.160.165.62', 80, False) * Connected to ('108.160.165.62', 80)

...

GET / HTTP/1.1 Host: dropbox.com

< HTTP/1.1 301 Moved Permanently < Server: nginx < Date: Fri, 18 Oct 2013 22:40:13 GMT < Content-Type: text/html < Content-Length: 178 < Connection: keep-alive < Location: https://dropbox.com/ < redirect to https://dropbox.com/ * Connecting to dropbox.com:443 using ssl * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, 199.47.217.179

Traceback (most recent call last): File "fetch3.py", line 211, in <module> main() File "fetch3.py", line 206, in main

body = loop.run_until_complete(fetch(**sys.argv[1], '-v' in sys.argv)) File "c:\Repos\tulip\asyncio\base_**events.py", line 177, in run_until_complete return future.result() File "c:\Repos\tulip\asyncio\**futures.py", line 221, in result raise self._exception File "c:\Repos\tulip\asyncio\tasks.**py", line 257, in _step result = coro.throw(exc) File "fetch3.py", line 192, in fetch yield from request.connect(pool) File "fetch3.py", line 80, in connect ssl=self.ssl) File "fetch3.py", line 36, in open_connection reader, writer = yield from open_connection(host, port, ssl=ssl) File "c:\Repos\tulip\asyncio\**streams.py", line 41, in open_connection lambda: protocol, host, port, **kwds) File "c:\Repos\tulip\asyncio\base_**events.py", line 356, in create_connection yield from waiter File "c:\Repos\tulip\asyncio\**futures.py", line 318, in __iter__ yield self # This tells Task to wait for completion. File "c:\Repos\tulip\asyncio\tasks.**py", line 308, in _wakeup value = future.result() File "c:\Repos\tulip\asyncio\**futures.py", line 221, in result raise self._exception File "c:\Repos\tulip\asyncio\**selector_events.py", line 579, in _on_handshake self._sock.do_handshake() File "C:\Repos\cpython-33\lib\ssl.**py", line 520, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553)

-- Richard

-- --Guido van Rossum (python.org/~guido)

On 19 October 2013 22:44, Christian Heimes <christian@python.org> wrote:

Am 19.10.2013 00:56, schrieb Guido van Rossum: A couple of months I had a long and fruitful discussion with MAL about the issue. Egenix PyOpenSSL installer comes with a root CA bundle. He tried a couple of approaches to handle trust settings with OpenSSL means. Eventually MAL had to split up the bundle into multiple files for each purpuse, see http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0....

We should *really* write a PEP about it, specify all details and get a proper review from real experts. This stuff is super complex and highly fragile. :(

At the very least, it would be good if you and/or MAL could review the cert verification in pip. PEP 453 makes that kinda important :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 20 October 2013 00:44, Christian Heimes <christian@python.org> wrote:

Am 19.10.2013 16:14, schrieb Nick Coghlan:

...

At the very least, it would be good if you and/or MAL could review the cert verification in pip. PEP 453 makes that kinda important :)

Where can I find the code for PEP 453?

It's the cert verification in pip that's relevant - the PEP was updated so that ensurepip itself never talks to the internet. So I guess that would mean checking the cert verification in pip's vendored copy of requests: https://github.com/pypa/pip/tree/develop/pip/vendor/requests (So I guess if you do find any issues, they would likely be applicable to the upstream requests package as well) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

One of the reasons we switched to using requests was to help centralize the SSL handling code over to requests. So any issues could be fixed there and we just pull in a newer version of requests. On Oct 19, 2013, at 11:52 AM, Christian Heimes <christian@python.org> wrote:

Signed PGP part Am 19.10.2013 16:59, schrieb Nick Coghlan:

...

It's the cert verification in pip that's relevant - the PEP was updated so that ensurepip itself never talks to the internet. So I guess that would mean checking the cert verification in pip's vendored copy of requests: https://github.com/pypa/pip/tree/develop/pip/vendor/requests

(So I guess if you do find any issues, they would likely be applicable to the upstream requests package as well)

Oh heck, where should I start?

The cacert.pem file is outdated. Also it's unclear who has generated the file and how it was generated from certdata.txt. It may very well contain revoked certificates, too. You can find the latest version of the file at

http://hg.mozilla.org/mozilla-central/file/tip/security/nss/lib/ckfw/builtin...

. A proper tool is required to generate a correct PEM file. It must understand *ALL* fields. I have some code for that but it's not ready yet.

pip uses requests and requests rolls its own code for or on top of Python stdlib modules, e.g. urllib3 with ssl_match_hostname. The method has the same security flaw as Python's ssl.match_hostname() function for IDNs. I'm a bit worried that we have to review and validate all 3rd party packages and copies of stdlib modules for issues.

The assert_fingerprint() function looks kinda funny. It uses sha1() or md5() on the DER representation of the cert. It's not how you are suppose to take fingerprints for cert pinning. But Python's ssl has no way to get the SPKI from the cert yet. I'm working on that as well.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 10/19/2013 12:46 PM, Ian Cordasco wrote:

Also the three of us maintaining requests and the author of urllib3 are all very conscious that the packaged pem file is outdated. We have an open issue about how to rebuild it accurately while taking into consideration (and not including) the ones that have been revoked. Any suggestions you have can be sent to me off list or reported on the issue tracker. Is this another issue like the time zone database? Something that needs to be packaged with some versions of Python, but that needs a mechanism to update it later for accuracy (which, in this case, also implies security)?

Could a similar mechanism be used for both?

On 20 October 2013 05:46, Ian Cordasco <graffatcolmingov@gmail.com> wrote:

Also the three of us maintaining requests and the author of urllib3 are all very conscious that the packaged pem file is outdated. We have an open issue about how to rebuild it accurately while taking into consideration (and not including) the ones that have been revoked. Any suggestions you have can be sent to me off list or reported on the issue tracker.

The requests issue Ian is referring to: https://github.com/kennethreitz/requests/issues/1659 The next version of PEP 453 will include getting this resolved as part of the integration timeline: ======================== * by December 29th (1 week prior to the scheduled date of 3.4.0 beta 2) ``requests`` certificate management issue resolved ``ensurepip`` updated to the final release of pip 1.5, or a subsequent maintenance release (including a suitably updated vendored copy of ``requests``) ======================== And also mentions it under the "security considerations" section for the bootstrapping mechanism: ======================== Only users that choose to use ``pip`` to communicate with PyPI will need to pay attention to the additional security considerations that come with doing so. However, the core CPython team will also assist with reviewing and resolving the `certificate update management issue <https://github.com/kennethreitz/requests/issues/1659>`__ currently affecting the ``requests`` project (and hence ``pip``). ======================== Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

You should ask Glyph too. He supplied lots of useful info about cert checking on the python-tulip list. On Sat, Oct 19, 2013 at 7:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 19 October 2013 22:44, Christian Heimes <christian@python.org> wrote:

...

Am 19.10.2013 00:56, schrieb Guido van Rossum: A couple of months I had a long and fruitful discussion with MAL about the issue. Egenix PyOpenSSL installer comes with a root CA bundle. He tried a couple of approaches to handle trust settings with OpenSSL means. Eventually MAL had to split up the bundle into multiple files for each purpuse, see

http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0....

...

We should *really* write a PEP about it, specify all details and get a proper review from real experts. This stuff is super complex and highly fragile. :(

At the very least, it would be good if you and/or MAL could review the cert verification in pip. PEP 453 makes that kinda important :)

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- --Guido van Rossum (python.org/~guido)