On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer <
xistence@0x58.com> wrote:
Hey all,
You may have seen some hub hub around psycopg2 and no longer shipping binary wheels by default [1][2] (and in fact using psycopg2-binary if you want wheels), and I wanted to bring it up here because it demonstrates a problem area with the current state of packaging in Python:
There is no good way for a new package to specify that it provides what another package would provide, and setuptools currently checks that all distributions are found before running the console scripts (so if a console script has a setup.py that depends on psycopg2 and you install psycopg2-binary it fails) [3]
So currently if you pip install psycopg2-binary and then install a project that uses psycopg2 as a dependency it will install psycopg2 over top of psycopg2-binary.
The author of psycopg2 stopped distributing binaries for psycopg2 because of issues with the version of SSL that Python was compiled for/what was used by psycopg2 and it causing all kinds of issues.
I don't have a proposal or a fix, but this is going to be an issue not just for psycopg2 but also for other projects that potentially distribute wheels that are built against a different version of OpenSSL.
I see two things that should get some thought:
1. How to have a package provide for another package (there are keywords but AFAIK they are currently ignored by pip)
2. How to handle/deal with shared libraries that are not versioned
The psycopg2 authors originally misdiagnosed the problem, and haven't
updated their docs since the problem was diagnosed further, so a lot
of people are confused about this whole psycopg2-binary thing :-(
There is no problem with shipping openssl in wheels. Lots of projects
do it fine. The reason psycopg2 is having problems is because of an
easily-fixable bug in psycopg2:
- Old versions of OpenSSL need some annoying configuration applied to
make them thread-safe
- libpq (which psycopg2 uses) normally does this configuration in one way
- the Python ssl module also normally does this configuration in a different way
- If libpq and the stdlib ssl module are both linked against the
*same* copy of openssl, then they can end up fighting with each other
- So the psycopg2 code has a special hack to unconditionally disable
libpq's thread-safety code, because the psycopg2 developers assumed
that psycopg2 and the stdlib ssl would *always* share the same copy of
openssl, and the stdlib ssl module would take care of the
thread-safety stuff
- Then they started distributing psycopg2 binaries with their own copy
of openssl in them, and of course they got crashes, because they've
turned off thread-safety, and now that they have their own copy of
openssl, no-one else is fixing it for them
So all they need to do to fix their wheels is either:
- somehow disable this patch in their wheel builds:
https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a816953a4Or else:
- switch to building their wheels against a newer version of openssl,
since newer versions of openssl are now thread-safe by default (thank
goodness)
Either way, it's totally fixable, and only the psycopg2 devs can fix it.
They've known this since July, but said they don't have energy to fix
it:
https://github.com/psycopg/psycopg2/issues/543#issuecomment-408352209I sympathize with that, but I wish they would tell people "hey, this
is a bug in psycopg2, we need help", instead of blaming python
packaging and trying to force all their downstreams to do wacky stuff
with dependencies to work around their bug.
It would indeed be nice if Python packages had better support for
Provides:, but psycopg2 is not really a good motivating use case.