Hi, On Sat, Apr 6, 2019 at 11:16 PM Bert JW Regeer <xistence@0x58.com> wrote:
On Apr 6, 2019, at 23:10, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer <xistence@0x58.com> wrote:
Hey all,
You may have seen some hub hub around psycopg2 and no longer shipping binary wheels by default [1][2] (and in fact using psycopg2-binary if you want wheels), and I wanted to bring it up here because it demonstrates a problem area with the current state of packaging in Python:
There is no good way for a new package to specify that it provides what another package would provide, and setuptools currently checks that all distributions are found before running the console scripts (so if a console script has a setup.py that depends on psycopg2 and you install psycopg2-binary it fails) [3]
So currently if you pip install psycopg2-binary and then install a project that uses psycopg2 as a dependency it will install psycopg2 over top of psycopg2-binary.
The author of psycopg2 stopped distributing binaries for psycopg2 because of issues with the version of SSL that Python was compiled for/what was used by psycopg2 and it causing all kinds of issues.
I don't have a proposal or a fix, but this is going to be an issue not just for psycopg2 but also for other projects that potentially distribute wheels that are built against a different version of OpenSSL.
I see two things that should get some thought:
1. How to have a package provide for another package (there are keywords but AFAIK they are currently ignored by pip) 2. How to handle/deal with shared libraries that are not versioned
The psycopg2 authors originally misdiagnosed the problem, and haven't updated their docs since the problem was diagnosed further, so a lot of people are confused about this whole psycopg2-binary thing :-(
There is no problem with shipping openssl in wheels. Lots of projects do it fine. The reason psycopg2 is having problems is because of an easily-fixable bug in psycopg2:
- Old versions of OpenSSL need some annoying configuration applied to make them thread-safe - libpq (which psycopg2 uses) normally does this configuration in one way - the Python ssl module also normally does this configuration in a different way - If libpq and the stdlib ssl module are both linked against the *same* copy of openssl, then they can end up fighting with each other - So the psycopg2 code has a special hack to unconditionally disable libpq's thread-safety code, because the psycopg2 developers assumed that psycopg2 and the stdlib ssl would *always* share the same copy of openssl, and the stdlib ssl module would take care of the thread-safety stuff - Then they started distributing psycopg2 binaries with their own copy of openssl in them, and of course they got crashes, because they've turned off thread-safety, and now that they have their own copy of openssl, no-one else is fixing it for them
So all they need to do to fix their wheels is either:
- somehow disable this patch in their wheel builds: https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a8...
Or else:
- switch to building their wheels against a newer version of openssl, since newer versions of openssl are now thread-safe by default (thank goodness)
Either way, it's totally fixable, and only the psycopg2 devs can fix it.
I don't work on psychopg2, nor do I understand the threading issues, but I offered to help, current status here: https://github.com/psycopg/psycopg2-wheels/pull/8 Cheers, Matthew