self.introduce(distutils-sig)
Hi all I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.) The main reason I want to get involved is the openly acknowledged lack of Windows expertise that's available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.) I expect I'll mainly be lurking until I can be useful, which is why I wanted to start with this post. I'm pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I'll be happy to clarify or research an answer. Cheers, Steve
On 2013-03-18 16:34:25 +0000, Steve Dower said:
Hi all I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.) The main reason I want to get involved is the openly acknowledged lack of Windows expertise that’s available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.) I expect I’ll mainly be lurking until I can be useful, which is why I wanted to start with this post. I’m pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I’ll be happy to clarify or research an answer.
Welcome!
Cheers, Steve _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Alex Clark · http://about.me/alex.clark
Welcome Steve! I really enjoyed your lightning talk (and would like to apologize for my audible "WHAT!?!" reaction when I heard the title as "Debugging Python with Microsoft Visual Studio" :) Welcome to the community, I've also been lurking here from some time, hoping to see where I can lend a hand in the packaging sub-community. --Dennis On Mon, Mar 18, 2013 at 2:26 PM, Alex Clark <aclark@aclark.net> wrote:
On 2013-03-18 16:34:25 +0000, Steve Dower said:
Hi all
I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.)
The main reason I want to get involved is the openly acknowledged lack of Windows expertise that’s available. I work at Microsoft and part of my job description is to contribute code/testing/time/**documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.)
I expect I’ll mainly be lurking until I can be useful, which is why I wanted to start with this post. I’m pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I’ll be happy to clarify or research an answer.
Welcome!
Cheers, Steve
______________________________**_________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/**mailman/listinfo/distutils-sig<http://mail.python.org/mailman/listinfo/distutils-sig>
-- Alex Clark · http://about.me/alex.clark
______________________________**_________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/**mailman/listinfo/distutils-sig<http://mail.python.org/mailman/listinfo/distutils-sig>
On Mon, Mar 18, 2013 at 9:34 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
I expect I’ll mainly be lurking until I can be useful, which is why I wanted to start with this post. I’m pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I’ll be happy to clarify or research an answer.
Great to hear and thanks :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Mar 18, 2013 at 12:34 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.)
That was a very cool demo.
The main reason I want to get involved is the openly acknowledged lack of Windows expertise that’s available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.)
I expect I’ll mainly be lurking until I can be useful, which is why I wanted to start with this post. I’m pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I’ll be happy to clarify or research an answer.
At the packaging panel, an issue was raised regarding issues with 32-bit and 64-bit windows packages. I don't remember the details. Were you there? If not, maybe someone can describe the issue here. Also an idea, fwiw: it would be awesome if MS provided something like travis-ci that executed tests on windows for open-source projects hosted in github (and other places like bitbucket, which I prefer). Maybe projects would start sporting "Windows: passing" buttons. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
On Mar 19, 2013, at 11:57 AM, Jim Fulton <jim@zope.com> wrote:
On Mon, Mar 18, 2013 at 12:34 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.)
That was a very cool demo.
The main reason I want to get involved is the openly acknowledged lack of Windows expertise that’s available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.)
I expect I’ll mainly be lurking until I can be useful, which is why I wanted to start with this post. I’m pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I’ll be happy to clarify or research an answer.
At the packaging panel, an issue was raised regarding issues with 32-bit and 64-bit windows packages. I don't remember the details. Were you there? If not, maybe someone can describe the issue here.
IIRC the issue is that the installers generated by distutils and friends are specific to either 32bit or 64bit windows. If I recall from my windows days 64bit windows puts it's "I'm installed here" info in the registry under a different location than 32bit installers. So When you attempt to use a 32bit installer it looks in the 32bit location, finds nothing and claims Python isn't installed. This is coupled with the fact if people publish installers at all for Windows it's typically 32bit only. I could of course be remembering wrong :)
Also an idea, fwiw: it would be awesome if MS provided something like travis-ci that executed tests on windows for open-source projects hosted in github (and other places like bitbucket, which I prefer). Maybe projects would start sporting "Windows: passing" buttons. :)
Especially if this worked _with_ travis :)
Jim
-- Jim Fulton http://www.linkedin.com/in/jimfulton _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
From: Jim Fulton On Mon, Mar 18, 2013 at 12:34 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.)
That was a very cool demo.
Thanks!
The main reason I want to get involved is the openly acknowledged lack of Windows expertise that's available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.)
I expect I'll mainly be lurking until I can be useful, which is why I wanted to start with this post. I'm pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I'll be happy to clarify or research an answer.
At the packaging panel, an issue was raised regarding issues with 32-bit and 64-bit windows packages. I don't remember the details. Were you there? If not, maybe someone can describe the issue here.
As I understand, the issue is the same as between different versions of Python and comes down to not being able to assume a compiler on Windows machines. It's easy to make a source file that will compile for any ABI and platform, but distributing binaries requires each one to be built separately. This doesn't have to be an onerous task - it can be scripted quite easily once you have all the required compilers - but it does take more effort than simply sharing a source file. Another issue is the CPython installer itself is very biased towards only having one of 32/64 and not both, despite having 4 possible configurations (excluding virtualenv and xcopy installs). The default installer settings will try and put 32 and 64 in the same folder, which is easily solved, but the registration information also goes into the same location. On Windows Vista and later (possibly XP, but I'm not going to promise that) there is automatic redirection for 32/64 that separates it, so that an installer will find the right path, but the per-user installs don't get this and installing both 32 and 64-bit versions will simply overwrite each other. As a result, it's hard for an MSI installer to find the right target version, and because it contains specific binaries finding the right version is essential. (I know so much about this because an IDE also has to find the versions, and there are simply some situations where it is impossible.)
Also an idea, fwiw: it would be awesome if MS provided something like travis-ci that executed tests on windows for open-source projects hosted in github (and other places like bitbucket, which I prefer). Maybe projects would start sporting "Windows: passing" buttons. :)
Bitbucket is starting to get some love here, and we've been pushing to get Mercurial on equal standing with Git internally. Right now, our small Python team isn't influential enough to get a commitment to a testing service, but there's absolutely no reason why one can't be set up with Windows VMs on any of the cloud services out there (pip is already using AWS for this). Funding from the PSF may be easier than funding from MS, though I've never tried to get funding from the PSF before so I could be wrong :)
Jim
-- Jim Fulton http://www.linkedin.com/in/jimfulton
On Tue, Mar 19, 2013 at 9:21 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Bitbucket is starting to get some love here, and we've been pushing to get Mercurial on equal standing with Git internally. Right now, our small Python team isn't influential enough to get a commitment to a testing service, but there's absolutely no reason why one can't be set up with Windows VMs on any of the cloud services out there (pip is already using AWS for this). Funding from the PSF may be easier than funding from MS, though I've never tried to get funding from the PSF before so I could be wrong :)
I use Shining Panda CI for my open source package testing, and they do offer Windows testing under Jenkins. However the free accounts aren't able to access the Windows buildbots - you need to pay for the Windows server time (which seems fair enough to me, given the difference in licensing costs between Debian and Windows, which are the two kinds of test environment they offer). The PSF offers funding for various things like Meetup.com fees for user groups, as well as funding for development sprints, so they could be interested in a program offering automated Windows testing through one of the CI services. The way to do that would be to put together a proposal for the board to consider, with a suggested budget and a mechanism for projects to apply for funding. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 March 2013 16:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
As I understand, the issue is the same as between different versions of Python and comes down to not being able to assume a compiler on Windows machines. It's easy to make a source file that will compile for any ABI and platform, but distributing binaries requires each one to be built separately. This doesn't have to be an onerous task - it can be scripted quite easily once you have all the required compilers - but it does take more effort than simply sharing a source file.
Another nice tool would be some sort of Windows build farm, where projects could submit a sdist and it would build wheels for a list of supported Python versions and architectures. That wouldn't work for projects with complex dependencies, obviously, but could cover a reasonable-sized chunk of PyPI (especially if dependencies could be added to the farm on request). And can I have a pony as well, of course... :-) Paul
On Wed, Mar 20, 2013 at 3:13 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 19 March 2013 16:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
As I understand, the issue is the same as between different versions of Python and comes down to not being able to assume a compiler on Windows machines. It's easy to make a source file that will compile for any ABI and platform, but distributing binaries requires each one to be built separately. This doesn't have to be an onerous task - it can be scripted quite easily once you have all the required compilers - but it does take more effort than simply sharing a source file.
Another nice tool would be some sort of Windows build farm, where projects could submit a sdist and it would build wheels for a list of supported Python versions and architectures. That wouldn't work for projects with complex dependencies, obviously, but could cover a reasonable-sized chunk of PyPI (especially if dependencies could be added to the farm on request).
And can I have a pony as well, of course... :-)
This also came up in the discussion over on http://simeonfranklin.com/blog/2013/mar/17/my-pycon-2013-poster/ I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/ (The security issues with that arrangement are non-trivial, but the convenience factor is huge) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/20/2013 04:42 PM, Nick Coghlan wrote:
On Wed, Mar 20, 2013 at 3:13 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 19 March 2013 16:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
As I understand, the issue is the same as between different versions of Python and comes down to not being able to assume a compiler on Windows machines. It's easy to make a source file that will compile for any ABI and platform, but distributing binaries requires each one to be built separately. This doesn't have to be an onerous task - it can be scripted quite easily once you have all the required compilers - but it does take more effort than simply sharing a source file.
Another nice tool would be some sort of Windows build farm, where projects could submit a sdist and it would build wheels for a list of supported Python versions and architectures. That wouldn't work for projects with complex dependencies, obviously, but could cover a reasonable-sized chunk of PyPI (especially if dependencies could be added to the farm on request).
And can I have a pony as well, of course... :-)
This also came up in the discussion over on http://simeonfranklin.com/blog/2013/mar/17/my-pycon-2013-poster/
I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/
(The security issues with that arrangement are non-trivial, but the convenience factor is huge)
Cheers, Nick.
Well a few other links: http://winbot.zope.org https://github.com/zopefoundation/zope.wineggbuilder https://github.com/zopefoundation/zope.winbot I can tell you getting such a beast to work takes quite some time. -- Best regards, Adam GROSZER -- Quote of the day: Each time you are honest and conduct yourself with honesty, a success force will drive you toward greater success. Each time you lie, even with a little white lie, there are strong forces pushing you toward failure. (Joseph Sugarman)
From: Nick Coghlan [mailto:ncoghlan@gmail.com] [snip]
I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/
(The security issues with that arrangement are non-trivial, but the convenience factor is huge)
FWIW, one of the guys on our team has met with Christoph and considers him trustworthy.
Cheers, Nick.
On Wed, Mar 20, 2013 at 9:03 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
From: Nick Coghlan [mailto:ncoghlan@gmail.com] [snip]
I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/
(The security issues with that arrangement are non-trivial, but the convenience factor is huge)
FWIW, one of the guys on our team has met with Christoph and considers him trustworthy.
Thanks, that's great to know, and ties into an idea that I just had. In addition to whether or not the build is trusted, there's also the risk of MITM attacks against the download site (less so when automated installers aren't involved, but still a risk). We just switched PyPI over to HTTPS for that very reason. The idle thought I had was that it may be useful if PyPI users could designate other users as "repackagers" for their project, and PyPI offered an interface that was *just* file uploads for an existing release. Then the pip developers, for example, could say "we trust Christoph to make our Windows installers", and grant him repackager access so he could upload the binaries for secure redistribution from PyPI rather than needing to host them himself. We'd probably want something like this for an effective build farm system anyway, this way it could work regardless of whether it was a human or an automated system converting the released sdists to platform specific binaries. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 20 March 2013 16:31, Nick Coghlan <ncoghlan@gmail.com> wrote:
Then the pip developers, for example, could say "we trust Christoph to make our Windows installers", and grant him repackager access so he could upload the binaries for secure redistribution from PyPI rather than needing to host them himself.
Another axis of the same idea would be to allow people to upload "unofficial" binaries. The individual would not need to be confirmed as trusted by the project, but his uploads would *not* be visible by default on PyPI. Users would be able to "opt in" to builds by that individual, and if they did, those builds would be merged in with what's on PyPI. That model is much closer to how Christoph is actually working at the moment - people can choose whether to trust him, but if they do they can get his builds and the upstream projects don't get involved. Paul
On Mar 20, 2013, at 12:45 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 20 March 2013 16:31, Nick Coghlan <ncoghlan@gmail.com> wrote:
Then the pip developers, for example, could say "we trust Christoph to make our Windows installers", and grant him repackager access so he could upload the binaries for secure redistribution from PyPI rather than needing to host them himself.
Another axis of the same idea would be to allow people to upload "unofficial" binaries. The individual would not need to be confirmed as trusted by the project, but his uploads would *not* be visible by default on PyPI. Users would be able to "opt in" to builds by that individual, and if they did, those builds would be merged in with what's on PyPI.
That model is much closer to how Christoph is actually working at the moment - people can choose whether to trust him, but if they do they can get his builds and the upstream projects don't get involved.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Why can't unofficial binaries just use a separate index? e.g. Christoph can just make an index with his binaries. This solution also works well if someone wants to maintain a curated PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 20 March 2013 18:29, Donald Stufft <donald@stufft.io> wrote:
Why can't unofficial binaries just use a separate index? e.g. Christoph can just make an index with his binaries.
This solution also works well if someone wants to maintain a curated PyPI.
The only real issue I know of is hosting. I've thought about doing this myself, but don't have (free) hosting space I could use, and I don't really feel like paying for and setting something up on spec. I could host the files somewhere like bitbucket, but that feels like an abuse for any substantial number of packages. I presume Christoph doesn't publish his binaries as an index because wininst installers are typically downloaded and installed manually.Although AIUI, easy_install could use them if they were in index format. But you're right, people *can* do that. Paul.
On Wed, Mar 20, 2013 at 3:10 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 20 March 2013 18:29, Donald Stufft <donald@stufft.io> wrote:
Why can't unofficial binaries just use a separate index? e.g. Christoph can just make an index with his binaries.
This solution also works well if someone wants to maintain a curated PyPI.
The only real issue I know of is hosting. I've thought about doing this myself, but don't have (free) hosting space I could use, and I don't really feel like paying for and setting something up on spec. I could host the files somewhere like bitbucket, but that feels like an abuse for any substantial number of packages.
I presume Christoph doesn't publish his binaries as an index because wininst installers are typically downloaded and installed manually.Although AIUI, easy_install could use them if they were in index format.
But you're right, people *can* do that. Paul.
If we know who to ask we can get hosting (not my area of expertise).
On Mar 20, 2013, at 12:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Mar 20, 2013 at 9:03 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
From: Nick Coghlan [mailto:ncoghlan@gmail.com] [snip]
I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/
(The security issues with that arrangement are non-trivial, but the convenience factor is huge)
FWIW, one of the guys on our team has met with Christoph and considers him trustworthy.
Thanks, that's great to know, and ties into an idea that I just had. In addition to whether or not the build is trusted, there's also the risk of MITM attacks against the download site (less so when automated installers aren't involved, but still a risk). We just switched PyPI over to HTTPS for that very reason.
The idle thought I had was that it may be useful if PyPI users could designate other users as "repackagers" for their project, and PyPI offered an interface that was *just* file uploads for an existing release.
I *think* if done properly a TUF secured API can be setup so as that you can delegate the role for signing certain files is delegated, but I'm not sure.
Then the pip developers, for example, could say "we trust Christoph to make our Windows installers", and grant him repackager access so he could upload the binaries for secure redistribution from PyPI rather than needing to host them himself.
We'd probably want something like this for an effective build farm system anyway, this way it could work regardless of whether it was a human or an automated system converting the released sdists to platform specific binaries.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/ (The security issues with that arrangement are non-trivial, but the convenience factor is huge) That webpage saved me a lot of headache with packages I was not able to build under Windows (with mingw64). I contacted Christoph if he could share buildscripts or somesuch, but got no reply from him (might be just too busy) - that build infrastructure would make it much easier to experiment with separate repositories, for personal/company/community needs, and perhaps evolve into some global repository of binaries for windows, with better security.
Cheers, Vaclav
Le 23 mars 2013 à 08:40, Václav Šmilauer <eu@doxos.eu> a écrit :
I was pointed to an interesting resource: http://www.lfd.uci.edu/~gohlke/pythonlibs/ (The security issues with that arrangement are non-trivial, but the convenience factor is huge) That webpage saved me a lot of headache with packages I was not able to build under Windows (with mingw64). I contacted Christoph if he could share buildscripts or somesuch, but got no reply from him (might be just too busy) - that build infrastructure would make it much easier to experiment with separate repositories, for personal/company/community needs, and perhaps evolve into some global repository of binaries for windows, with better security.
Dis you try www.pythonxy.com ?
Cheers, Vaclav
François
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/20/2013 06:13 AM, Paul Moore wrote:
Another nice tool would be some sort of Windows build farm, where projects could submit a sdist and it would build wheels for a list of supported Python versions and architectures. That wouldn't work for projects with complex dependencies, obviously, but could cover a reasonable-sized chunk of PyPI (especially if dependencies could be added to the farm on request).
The Zope Foundation pays hosting charges for a box which runs Windows tests for ZF projects, and also builds and uploads Windows binaries (eggs and MSIs) for them when they are released. http://winbot.zope.org/ As an example, look at any recent zope.interface release, e.g.: https://pypi.python.org/pypi/zope.interface/4.0.5#downloads Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFKAP0ACgkQ+gerLs4ltQ6UZwCgkxfOtrArGn/F5dKPk6+QepWV 7jYAniBreYijRKhevNS6rDUteePNzfZW =m0LP -----END PGP SIGNATURE-----
participants (12)
-
Adam GROSZER -
Alex Clark -
Daniel Holth -
Dennis Coldwell -
Donald Stufft -
Francois Chenais -
Jim Fulton -
Nick Coghlan -
Paul Moore -
Steve Dower -
Tres Seaver -
Václav Šmilauer