[Twisted-Python] [RFC] Introducing six as a dependency
Hello all, as part of my work on porting Twisted to Python 3, I have considered using the six library[1] to help with some issues. six is basically a compatibility library - same idea as our twisted.python.compat module. Now, one approach is to add six as a dependency of Twisted - it is a very small library so hardly a problem; on the other hand, it is an additional dependency. The other approach would be to copy the required code over to the t.p.compat module (six' license[2] is basically "do what you want with this code"), but this is additional work and we might miss out on the eventual bugfix (though there has only been a couple since the project started). In general, I have heard good things about six. As a note, currently, I would need at least the reraise and exec_ functions from six; additionally, a lot of the things from the six.moves package are useful. And, of course, there might be other compatibility functions which I'd need, I just haven't encountered yet (I do only use these if absolutely required, though). What do you all think? In the end, it boils down to "additional dependency" vs. "less code to maintain". Personally, I could go either way - using six makes things slightly easier, but I could just copy over the code we need and be done with it. [1] http://packages.python.org/six/ [2] https://bitbucket.org/gutworth/six/src/d81f633c45dd/LICENSE -- Vladimir Perić
On Thu, Jul 26, 2012 at 4:55 PM, Vladimir Perić <vlada.peric@gmail.com> wrote:
Hello all,
Hi! ...
Now, one approach is to add six as a dependency of Twisted - it is a very small library so hardly a problem; on the other hand, it is an additional dependency. ... What do you all think? In the end, it boils down to "additional dependency" vs. "less code to maintain".
Not strictly what I think, but here's the relevant bits from what Glyph said last time I asked about adding a dependency, testtools, to Twisted: """ Users still routinely struggle with the one dependency we allowed Twisted core to have - zope.interface. I do still think that's worth it, since it freed us from a significant and complex maintenance burden. And I do sometimes wish that we could make it an optional or bundled dependency, to give users who have to download Twisted themselves a gentler on-ramp. [...] [...] I would set the bar very high for making testtools a required dependency for Twisted's own test suite. Just for starters, the Python packaging ecosystem disaster would need to be fixed; also, the name of the package should be changed to be more unique so that users wouldn't find things like <http://www.testtools.com/> and <https://github.com/ferruhy/testtools> when searching around the web for the contents of the inevitable packaging error message. """ jml
Does Glyph still believe these arguments to be true today? I don't know how long ago he said that, but I do know there's a lot of stuff that's been done to make packaging better :) Perhaps we could look into shipping twisted releases with and without dependencies? I personally have no such issues since I just let pip/tox handle everything for me and it figures it out already. cheers lvh On 26 Jul 2012, at 18:02, Jonathan Lange <jml@mumak.net> wrote:
On Thu, Jul 26, 2012 at 4:55 PM, Vladimir Perić <vlada.peric@gmail.com> wrote:
Hello all,
Hi!
...
Now, one approach is to add six as a dependency of Twisted - it is a very small library so hardly a problem; on the other hand, it is an additional dependency. ... What do you all think? In the end, it boils down to "additional dependency" vs. "less code to maintain".
Not strictly what I think, but here's the relevant bits from what Glyph said last time I asked about adding a dependency, testtools, to Twisted:
""" Users still routinely struggle with the one dependency we allowed Twisted core to have - zope.interface. I do still think that's worth it, since it freed us from a significant and complex maintenance burden. And I do sometimes wish that we could make it an optional or bundled dependency, to give users who have to download Twisted themselves a gentler on-ramp. [...]
[...] I would set the bar very high for making testtools a required dependency for Twisted's own test suite. Just for starters, the Python packaging ecosystem disaster would need to be fixed; also, the name of the package should be changed to be more unique so that users wouldn't find things like <http://www.testtools.com/> and <https://github.com/ferruhy/testtools> when searching around the web for the contents of the inevitable packaging error message. """
jml
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, Jul 26, 2012 at 5:22 PM, Laurens Van Houtven <_@lvh.cc> wrote:
Does Glyph still believe these arguments to be true today? I don't know how long ago he said that, but I do know there's a lot of stuff that's been done to make packaging better :)
Sorry, should have dated: Feb 6, 2012. jml
Le Jul 26, 2012 à 9:22 AM, Laurens Van Houtven <_@lvh.cc> a écrit :
Perhaps we could look into shipping twisted releases with and without dependencies? I personally have no such issues since I just let pip/tox handle everything for me and it figures it out already.
This is only accurate to within a first approximation, but... pip does not work for Windows users. And easy_install there comes with its own set of warts. So there's no good answer. If you have a good proposal for how to ship a sumo Twisted which bundles all its dependencies in a way that would actually reach the users who encounter dependency management problems, that would be great. I think it would be worthwhile to discuss it privately first though, since there are enough publicly-archived flamewars about Python packaging technology ;-). -glyph
On Thu, Jul 26, 2012 at 12:48 PM, Glyph <glyph@twistedmatrix.com> wrote:
Le Jul 26, 2012 à 9:22 AM, Laurens Van Houtven <_@lvh.cc> a écrit :
Perhaps we could look into shipping twisted releases with and without dependencies? I personally have no such issues since I just let pip/tox handle everything for me and it figures it out already.
This is only accurate to within a first approximation, but... pip does not work for Windows users.
Um, howso? I use it all the time, including for installing Twisted. Unless you mean it doesn't work fro installing twisted for those who don't have a C compiler, which I guess could be true. I always have one installed, so I haven't tried without one. Kevin Horn
On Thu, Jul 26, 2012 at 3:06 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
This is only accurate to within a first approximation, but... pip does not
work for Windows users.
Um, howso? I use it all the time, including for installing Twisted. Unless you mean it doesn't work fro installing twisted for those who don't have a C compiler, which I guess could be true. I always have one installed, so I haven't tried without one.
My attempt to set things up on Windows did not go well, at all. Getting a compiler going was IIRC non-trivial.
Le Jul 26, 2012 à 12:06 PM, Kevin Horn <kevin.horn@gmail.com> a écrit :
... for those who don't have a C compiler ...
AKA "windows users". Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org builds use and you don't have to go trawling through python-dev archives to figure it out...) -glyph
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org builds use and you don't have to go trawling through python-dev archives to figure it out...)
Hi, the python source code comes with a directory containing MSVC project files, named after the version of Visual Studio product: http://docs.python.org/using/windows.html#compiling-python-on-windows
On 03:31 am, anis.mourad@gmail.com wrote:
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org builds use and you don't have to go trawling through python-dev archives to figure it out...)
Hi, the python source code comes with a directory containing MSVC project files, named after the version of Visual Studio product: http://docs.python.org/using/windows.html#compiling-python-on-windows
Which means if you want to use pip to install Twisted, all you need is the C compiler specified by the source for the version of Python you're building, inferred after making the logical leap that the newest version of MSVC with a corresponding entry in the PC directory of that Python source is probably the one that was used to build the official release. And that's why the idea that pip is unusable on Windows has some traction. Jean-Paul
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org >> builds use and you don't have to go trawling through python-dev archives to figure it out...)
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008. - VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler) To build Twisted working with official binary Python builds, the exact same compiler is needed: VS2008.
Why are compilers and MSVC++ relevant to six as a new dependency? It consists of one .py file. On Fri, Jul 27, 2012 at 7:32 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org >> builds use and you don't have to go trawling through python-dev archives to figure it out...)
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008.
- VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler)
To build Twisted working with official binary Python builds, the exact same compiler is needed: VS2008.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Jasper
Am 27.07.2012 12:32, schrieb Tobias Oberstein:
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org >> builds use and you don't have to go trawling through python-dev archives to figure it out...)
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008.
- VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler) But you can add the cmdline 64-bit compiler by downloading the free Windows SDK.
Michael -- Michael Schlenker Software Architect CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen http://www.contact.de/ E-Mail: msc@contact.de Sitz der Gesellschaft: Bremen Geschäftsführer: Karl Heinz Zachries, Ralf Holtgrefe Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215
On 10:32 am, tobias.oberstein@tavendo.de wrote:
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org >> builds use and you don't have to go trawling through python-dev archives to figure it out...)
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008.
- VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler)
To build Twisted working with official binary Python builds, the exact same compiler is needed: VS2008.
The question isn't "What compiler is needed to build Twisted?" People on this list probably know that already (I certainly do, since I set up several of Twisted's Windows buildslaves). The question is "How could a user ever hope to find out what compiler is needed to build Twisted?" As long as the Python documentation fails to answer this question better than it does now (and VS2008 is not the correct answer: the correct answer _depends on the Python version_), Windows users will not have an easy time of it. Jean-Paul
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008.
- VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler)
To build Twisted working with official binary Python builds, the exact same compiler is needed: VS2008.
The question isn't "What compiler is needed to build Twisted?" People on this list probably know that already (I certainly do, since I set up several of Twisted's Windows buildslaves). The question is "How could a user ever hope to find out what compiler is needed to build Twisted?"
IMO, the build should figure that out (pip/easy_install). Why should I take the burden of searching through docs (however easy those may be or may not be to find), when the build tool can detect and inform me about the compiler needed? Of course that again underlines the deficiencies glyph pointed out with pip/easy_install. Twisted might compensate those by including detection in it's build scripts .. but that will be controversial for sure (compensating for stuff that would better be fixed elsewhere ..)
As long as the Python documentation fails to answer this question better than it does now (and VS2008 is not the correct answer: the correct answer _depends on the Python version_), Windows users will not have an easy time of it.
Yeah, right;) I didn't want to say Python 1.5 builds with VS2008, but the current Py2 one.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Fri, Jul 27, 2012 at 9:40 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008.
- VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler)
To build Twisted working with official binary Python builds, the exact same compiler is needed: VS2008.
The question isn't "What compiler is needed to build Twisted?" People on this list probably know that already (I certainly do, since I set up several of Twisted's Windows buildslaves). The question is "How could a user ever hope to find out what compiler is needed to build Twisted?"
IMO, the build should figure that out (pip/easy_install). Why should I take the burden of searching through docs (however easy those may be or may not be to find), when the build tool can detect and inform me about the compiler needed?
Of course that again underlines the deficiencies glyph pointed out with pip/easy_install.
Twisted might compensate those by including detection in it's build scripts .. but that will be controversial for sure (compensating for stuff that would better be fixed elsewhere ..)
As long as the Python documentation fails to answer this question better
than it
does now (and VS2008 is not the correct answer: the correct answer _depends on the Python version_), Windows users will not have an easy time of it.
Yeah, right;) I didn't want to say Python 1.5 builds with VS2008, but the current Py2 one.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Actually, you can use VS 2010 with python but you need to modify sth. deploying python app (not only with twisted) on windows is troublesome. Sometimes, I have to manually download .msi files and execute it order by order. I am sure that 99% of windows servers don't have a working compiler no matter it is "free" or not. Regards gelin yan
On Fri, Jul 27, 2012 at 6:59 AM, <exarkun@twistedmatrix.com> wrote:
On 10:32 am, tobias.oberstein@tavendo.de wrote:
Even having full access to MSDN, it can be incredibly obscure to discover which Python version goes with which Visual Studio product. (Someone, please prove me wrong and indicate that there's a web page that shows what the official python.org >> builds use and you don't have to go trawling through python-dev archives to figure it out...)
It is obscure. And I don't have a link, but the official Python on Windows builds are done using Microsoft Visual C++ 2008.
- VS 2010 will NOT work - the free VS 2008 Express works (for 32-bit builds .. it does not include a 64-bit compiler)
To build Twisted working with official binary Python builds, the exact same compiler is needed: VS2008.
The question isn't "What compiler is needed to build Twisted?" People on this list probably know that already (I certainly do, since I set up several of Twisted's Windows buildslaves). The question is "How could a user ever hope to find out what compiler is needed to build Twisted?"
As long as the Python documentation fails to answer this question better than it does now (and VS2008 is not the correct answer: the correct answer _depends on the Python version_), Windows users will not have an easy time of it.
For the benefit of those who may come across this thread in the future, I've been using MinGW to build Twisted (on WinXP and Win7) for several years, with no problems at all. Kevin Horn
On Thu, Jul 26, 2012 at 10:55 AM, Vladimir Perić <vlada.peric@gmail.com> wrote:
Hello all,
as part of my work on porting Twisted to Python 3, I have considered using the six library[1] to help with some issues. six is basically a compatibility library - same idea as our twisted.python.compat module. Now, one approach is to add six as a dependency of Twisted - it is a very small library so hardly a problem; on the other hand, it is an additional dependency. The other approach would be to copy the required code over to the t.p.compat module (six' license[2] is basically "do what you want with this code"), but this is additional work and we might miss out on the eventual bugfix (though there has only been a couple since the project started). In general, I have heard good things about six.
As a note, currently, I would need at least the reraise and exec_ functions from six; additionally, a lot of the things from the six.moves package are useful. And, of course, there might be other compatibility functions which I'd need, I just haven't encountered yet (I do only use these if absolutely required, though).
What do you all think? In the end, it boils down to "additional dependency" vs. "less code to maintain".
Personally, I could go either way - using six makes things slightly easier, but I could just copy over the code we need and be done with it.
FWIW, I'm +1 on copying the bits we need into Twisted and +0 on adding a new dep. d
[1] http://packages.python.org/six/ [2] https://bitbucket.org/gutworth/six/src/d81f633c45dd/LICENSE -- Vladimir Perić
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Le Jul 26, 2012 à 9:21 AM, Duncan McGreggor <oubiwann@twistedmatrix.com> a écrit :
FWIW, I'm +1 on copying the bits we need into Twisted and +0 on adding a new dep.
I'd be closer to +0 on copying the bits we need (I'm not excited about it, but it seems to be necessary) and -1 on adding them as a new dependency, for the reasons that jml already cited. -glyph
On 26 July 2012 16:55, Vladimir Perić <vlada.peric@gmail.com> wrote: ...
As a note, currently, I would need at least the reraise and exec_ functions from six;
Fwiw, reraise is fairly trivial and wouldn't be a big thing to carry in Twisted. exec(code, [globals, [locals]]) works on both Python 2 and Python 3, and seem to have similar behaviour, so I'm not sure why exec_ is needed. Indeed, test_exec_() from the six source passes using the builtin exec in place of six.exec_. Ah, I've tried only with Python 2.6 and 2.7; perhaps there are differences on earlier 2.x versions?
On 7/26/12, Gavin Panella <gavin@gromper.net> wrote:
On 26 July 2012 16:55, Vladimir Perić <vlada.peric@gmail.com> wrote: ...
As a note, currently, I would need at least the reraise and exec_ functions from six;
Fwiw, reraise is fairly trivial and wouldn't be a big thing to carry in Twisted.
exec(code, [globals, [locals]]) works on both Python 2 and Python 3, and seem to have similar behaviour, so I'm not sure why exec_ is needed. Indeed, test_exec_() from the six source passes using the builtin exec in place of six.exec_. Ah, I've tried only with Python 2.6 and 2.7; perhaps there are differences on earlier 2.x versions?
Actually, Twisted now only supports 2.6 and 2.7 anyway, the issue is that the old syntax "exec code in globs, locs" is a SyntaxError in Python 3. In any case, in light of the discussions (and especially Jonathan's repost of Glyph's opinion), I'll opt for copying the required code to our twisted.python.compat module (copying the whole of six is not really practical, particularly because we'd still need additional compatibility modules). Thanks all for giving your input.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Vladimir Perić
On Thu, 26 Jul 2012 17:55:50 +0200 Vladimir Perić <vlada.peric@gmail.com> wrote:
Hello all,
as part of my work on porting Twisted to Python 3, I have considered using the six library[1] to help with some issues. six is basically a compatibility library - same idea as our twisted.python.compat module. Now, one approach is to add six as a dependency of Twisted - it is a very small library so hardly a problem; on the other hand, it is an additional dependency. The other approach would be to copy the required code over to the t.p.compat module (six' license[2] is basically "do what you want with this code"), but this is additional work and we might miss out on the eventual bugfix (though there has only been a couple since the project started).
Copying them sounds like the saner approach to me. These compatibility wrappers are unlikely to change significantly, and requiring a dependency on such a small module is not very helpful. If you really want to minimize maintenance, you can also ship six as a twisted.python submodule, and synchronize the copy when there are interesting bug fixes. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
participants (15)
-
Antoine Pitrou
-
Duncan McGreggor
-
exarkun@twistedmatrix.com
-
Gavin Panella
-
gelin yan
-
Glyph
-
Itamar Turner-Trauring
-
Jasper St. Pierre
-
Jonathan Lange
-
Kevin Horn
-
Laurens Van Houtven
-
Michael Schlenker
-
Mourad Anis
-
Tobias Oberstein
-
Vladimir Perić