multiple backports of ipaddress and a world of pain
Hi All, (Apologies for copying in the maintainers of the two backports and django-netfields directly, I'm not sure you're on this distutils list...) This is painful and horrible, and I wish pip would prevent modules/packages with the same name being installed by different distributions at the same time, but even if it did, that would just force something to happen rather than this: So, RHEL7, for worse or worse, ships with Python 2.7.5. That means to keep pip happy, you need to do these dances in all the virtualenvs you create: http://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarni... http://urllib3.readthedocs.org/en/latest/security.html#pyopenssl One of those extra packages drags in this backport: https://pypi.python.org/pypi/ipaddress Yay! Now we have a happy pip talking to both PyPI and our internal DevPI server! Right, so in a Django project I need to use https://pypi.python.org/pypi/django-netfields. This, however, chooses this backport instead: https://pypi.python.org/pypi/py2-ipaddress So, now we have two packages installing ipaddress.py, except they're two very different versions and make different assumptions about what to do with Python 2 strings. What should happen here? (other than me crying a lot...) Chris
Code that uses py2-ipaddress will break upon migrating to Python 3, and potentially in really subtle ways. For instance, import py2_ipaddress as ipaddress ipaddress.ip_address(b'\x3a\x3a\x31\x32') ipaddress.ip_address(open('file', 'rb').read(4)) has different semantics in Python 2 and Python 3. Also note that if you actually want to generate an ipaddress object from a binary representation, py2-ipaddress' "solution" ipaddress.ip_address(bytearray(b'\xff\x00\x00\x01')) will break as well under Python 3, but at least it will throw an exception and not silently do something different. Therefore, code that uses py2-ipaddress needs to be fixed anyways in order to work correctly under Python 3 - might as well do it now. py2-ipaddress' API is incompatible with the ipaddress from the stdlib, so I don't think it should claim the module name ipaddress in the first place. Why one would actively introduce incompatibilities between Python 2 and Python 3 *after Python 3 has long been released* is beyond my understanding anyways. Specifically for django-netfields, a workaround is to always use character strings (unicode type in Python 2, str in Python 3). Greetings from Düsseldorf, Philipp On 16.02.2016 19:37, Chris Withers wrote:
Hi All,
(Apologies for copying in the maintainers of the two backports and django-netfields directly, I'm not sure you're on this distutils list...)
This is painful and horrible, and I wish pip would prevent modules/packages with the same name being installed by different distributions at the same time, but even if it did, that would just force something to happen rather than this:
So, RHEL7, for worse or worse, ships with Python 2.7.5. That means to keep pip happy, you need to do these dances in all the virtualenvs you create:
http://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarni...
http://urllib3.readthedocs.org/en/latest/security.html#pyopenssl
One of those extra packages drags in this backport:
https://pypi.python.org/pypi/ipaddress
Yay! Now we have a happy pip talking to both PyPI and our internal DevPI server!
Right, so in a Django project I need to use https://pypi.python.org/pypi/django-netfields. This, however, chooses this backport instead:
https://pypi.python.org/pypi/py2-ipaddress
So, now we have two packages installing ipaddress.py, except they're two very different versions and make different assumptions about what to do with Python 2 strings.
What should happen here? (other than me crying a lot...)
Chris
On 16/02/2016 22:31, Philipp Hagemeister wrote:
Code that uses py2-ipaddress will break upon migrating to Python 3, and potentially in really subtle ways. For instance,
import py2_ipaddress as ipaddress ipaddress.ip_address(b'\x3a\x3a\x31\x32') ipaddress.ip_address(open('file', 'rb').read(4))
has different semantics in Python 2 and Python 3. Also note that if you actually want to generate an ipaddress object from a binary representation, py2-ipaddress' "solution" ipaddress.ip_address(bytearray(b'\xff\x00\x00\x01')) will break as well under Python 3, but at least it will throw an exception and not silently do something different.
Therefore, code that uses py2-ipaddress needs to be fixed anyways in order to work correctly under Python 3 - might as well do it now.
Indeed, James has thankfully switched django-netfields to ipaddress...
py2-ipaddress' API is incompatible with the ipaddress from the stdlib, so I don't think it should claim the module name ipaddress in the first place. Why one would actively introduce incompatibilities between Python 2 and Python 3 *after Python 3 has long been released* is beyond my understanding anyways.
Specifically for django-netfields, a workaround is to always use character strings (unicode type in Python 2, str in Python 3).
The code James uses as a result isn't that pretty, multiple occurrences of: try: value = unicode(value) except NameError: pass What would you recommend instead? When I monkeypatched this yesterday, I went with: if isinstance(value, bytes): value = value.decode('ascii') ...but I wonder if there's something better? (The context here is data coming back from Postgres, which is always str in Python 2, and only contains ip or netmask, so really shouldn't have anything non-ascii in it!) cheers, Chris
On 17.02.2016 08:51, Chris Withers wrote:
The code James uses as a result isn't that pretty, multiple occurrences of:
try: value = unicode(value) except NameError: pass
This is the last resort when you truly don't know what the input is, and you are sure the string should only contain ASCII characters. Works fine for ipaddress purposes.
What would you recommend instead? When I monkeypatched this yesterday, I went with:
if isinstance(value, bytes): value = value.decode('ascii')
...but I wonder if there's something better?
Either is fine by me. Preferably, one should go fix the source to return character strings in the first place. Usually, this includes from __future__ import unicode_literals, io.open with an encoding parameter, and inserting unconditional decode calls when the source really is bytes. I don't know enough about Python's postgres connector to give a better solution. How does python-postgres handle character strings when they're not all ASCII? If it's returning b'D\xc3\xbcsseldorf' on 2.x and u'Düsseldorf' on 3.x, then you have to conditionally decode('utf-8') all strings anyways in order to work on Python 3 as well as 2.x, haven't you? Greetings from Düsseldorf, Philipp
On Wednesday, February 17 12:30, Philipp Hagemeister wrote:
Preferably, one should go fix the source to return character strings in the first place. Usually, this includes from __future__ import unicode_literals, io.open with an encoding parameter, and inserting unconditional decode calls when the source really is bytes.
The code in question is Django, and it seems to be good in the currently supported version. This issue only occurred in older versions. It actually surprised me that this occurred, since Django is usually quite good at ensuring unicode strings throughout the code. -- James Oakley jfunk@funktronics.ca
On 17 February 2016 at 04:37, Chris Withers <chris@simplistix.co.uk> wrote:
Hi All,
(Apologies for copying in the maintainers of the two backports and django-netfields directly, I'm not sure you're on this distutils list...)
This is painful and horrible, and I wish pip would prevent modules/packages with the same name being installed by different distributions at the same time, but even if it did, that would just force something to happen rather than this:
So, RHEL7, for worse or worse, ships with Python 2.7.5.
It's 2.7.5 + important security backports, so any package that relies on PEP 466 features like ssl.create_default_context() should be fine in 7.2+. (You can also switch on default certificate verification if you want it: https://access.redhat.com/articles/2039753 )
That means to keep pip happy, you need to do these dances in all the virtualenvs you create:
http://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarni... http://urllib3.readthedocs.org/en/latest/security.html#pyopenssl
If urllib3 is actually using version detection rather than feature detection as recommended in PEP 466 ( https://www.python.org/dev/peps/pep-0466/#backwards-compatibility-considerat...), then that's a missing bug report against urllib3
One of those extra packages drags in this backport:
https://pypi.python.org/pypi/ipaddress
Yay! Now we have a happy pip talking to both PyPI and our internal DevPI server!
Right, so in a Django project I need to use https://pypi.python.org/pypi/django-netfields. This, however, chooses this backport instead:
https://pypi.python.org/pypi/py2-ipaddress
So, now we have two packages installing ipaddress.py, except they're two very different versions and make different assumptions about what to do with Python 2 strings.
What should happen here? (other than me crying a lot...)
It looks like you found a resolution to this part of the problem, but those dependencies should only be needed on 7.0 and 7.1 Unfortunately, I missed this use case when PEP 508 was being defined, so there's currently no capability for Python level dependencies to be conditional on the presence or absence of particular attributes in other modules :( Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 February 2016 at 16:59, Chris Withers <chris@simplistix.co.uk> wrote:
Hi Nick,
On 18/02/2016 13:32, Nick Coghlan wrote:
On 17 February 2016 at 04:37, Chris Withers <chris@simplistix.co.uk> wrote:
So, RHEL7, for worse or worse, ships with Python 2.7.5.
It's 2.7.5 + important security backports, so any package that relies on PEP 466 features like ssl.create_default_context() should be fine in 7.2+. (You can also switch on default certificate verification if you want it: https://access.redhat.com/articles/2039753 )
The company I'm with at the moment is one of the more aggressive operating system release followers I've worked with or for, and even we're not on 7.2 yet!
Aye, being on 7.x at all already means they're doing better than a lot of folks. If fast adoption of new distro versions was entirely typical we wouldn't still be having to encourage people to stop running their own applications in the RHEL 6 system Python :)
<multiple ipaddress backports>
It looks like you found a resolution to this part of the problem, but those dependencies should only be needed on 7.0 and 7.1
Unfortunately, I missed this use case when PEP 508 was being defined, so there's currently no capability for Python level dependencies to be conditional on the presence or absence of particular attributes in other modules :(
Not sure that's such a biggie here, I'd more like to see pip at least notice that it's trying to install two files into the same location.
It's mainly a note to myself that there's a current gap in our backporting story there, since it means folks *can't* currently do ssl module feature detection at installation time to decide if they need PyOpenSSL as a dependency or not. At the moment that only impacts RHEL 7.2+, but if PEP 493 gets accepted, the omission may end up affecting other LTS Linux distros as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Chris Withers -
James Oakley -
Nick Coghlan -
Philipp Hagemeister