On Apr 6, 2019, at 23:10, Nathaniel Smith wrote:

On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer wrote:

...

Hey all,

You may have seen some hub hub around psycopg2 and no longer shipping binary wheels by default [1][2] (and in fact using psycopg2-binary if you want wheels), and I wanted to bring it up here because it demonstrates a problem area with the current state of packaging in Python:

There is no good way for a new package to specify that it provides what another package would provide, and setuptools currently checks that all distributions are found before running the console scripts (so if a console script has a setup.py that depends on psycopg2 and you install psycopg2-binary it fails) [3]

So currently if you pip install psycopg2-binary and then install a project that uses psycopg2 as a dependency it will install psycopg2 over top of psycopg2-binary.

The author of psycopg2 stopped distributing binaries for psycopg2 because of issues with the version of SSL that Python was compiled for/what was used by psycopg2 and it causing all kinds of issues.

I don't have a proposal or a fix, but this is going to be an issue not just for psycopg2 but also for other projects that potentially distribute wheels that are built against a different version of OpenSSL.

I see two things that should get some thought:

1. How to have a package provide for another package (there are keywords but AFAIK they are currently ignored by pip) 2. How to handle/deal with shared libraries that are not versioned

The psycopg2 authors originally misdiagnosed the problem, and haven't updated their docs since the problem was diagnosed further, so a lot of people are confused about this whole psycopg2-binary thing :-(

There is no problem with shipping openssl in wheels. Lots of projects do it fine. The reason psycopg2 is having problems is because of an easily-fixable bug in psycopg2:

- Old versions of OpenSSL need some annoying configuration applied to make them thread-safe - libpq (which psycopg2 uses) normally does this configuration in one way - the Python ssl module also normally does this configuration in a different way - If libpq and the stdlib ssl module are both linked against the *same* copy of openssl, then they can end up fighting with each other - So the psycopg2 code has a special hack to unconditionally disable libpq's thread-safety code, because the psycopg2 developers assumed that psycopg2 and the stdlib ssl would *always* share the same copy of openssl, and the stdlib ssl module would take care of the thread-safety stuff - Then they started distributing psycopg2 binaries with their own copy of openssl in them, and of course they got crashes, because they've turned off thread-safety, and now that they have their own copy of openssl, no-one else is fixing it for them

So all they need to do to fix their wheels is either:

- somehow disable this patch in their wheel builds: https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a8...

Or else:

- switch to building their wheels against a newer version of openssl, since newer versions of openssl are now thread-safe by default (thank goodness)

Either way, it's totally fixable, and only the psycopg2 devs can fix it.

They've known this since July, but said they don't have energy to fix it: https://github.com/psycopg/psycopg2/issues/543#issuecomment-408352209

I sympathize with that, but I wish they would tell people "hey, this is a bug in psycopg2, we need help", instead of blaming python packaging and trying to force all their downstreams to do wacky stuff with dependencies to work around their bug.

It would indeed be nice if Python packages had better support for Provides:, but psycopg2 is not really a good motivating use case.

No, it is not a good motivating use case, but it is an issue that exists _today_ hence why I brought it up.

-n

-- Nathaniel J. Smith -- https://vorpus.org https://vorpus.org/

Thank you for your information Nathaniel. It is a shame that this fix was not used so that they could continue to ship psycopg2 wheels. Sadly the maintainers have made more work for themselves by not following up on your suggestions :-/ Bert

Hi, On Sat, Apr 6, 2019 at 11:16 PM Bert JW Regeer wrote:

On Apr 6, 2019, at 23:10, Nathaniel Smith wrote:

On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer wrote:

Hey all,

You may have seen some hub hub around psycopg2 and no longer shipping binary wheels by default [1][2] (and in fact using psycopg2-binary if you want wheels), and I wanted to bring it up here because it demonstrates a problem area with the current state of packaging in Python:

There is no good way for a new package to specify that it provides what another package would provide, and setuptools currently checks that all distributions are found before running the console scripts (so if a console script has a setup.py that depends on psycopg2 and you install psycopg2-binary it fails) [3]

So currently if you pip install psycopg2-binary and then install a project that uses psycopg2 as a dependency it will install psycopg2 over top of psycopg2-binary.

The author of psycopg2 stopped distributing binaries for psycopg2 because of issues with the version of SSL that Python was compiled for/what was used by psycopg2 and it causing all kinds of issues.

I don't have a proposal or a fix, but this is going to be an issue not just for psycopg2 but also for other projects that potentially distribute wheels that are built against a different version of OpenSSL.

I see two things that should get some thought:

1. How to have a package provide for another package (there are keywords but AFAIK they are currently ignored by pip) 2. How to handle/deal with shared libraries that are not versioned

The psycopg2 authors originally misdiagnosed the problem, and haven't updated their docs since the problem was diagnosed further, so a lot of people are confused about this whole psycopg2-binary thing :-(

There is no problem with shipping openssl in wheels. Lots of projects do it fine. The reason psycopg2 is having problems is because of an easily-fixable bug in psycopg2:

- Old versions of OpenSSL need some annoying configuration applied to make them thread-safe - libpq (which psycopg2 uses) normally does this configuration in one way - the Python ssl module also normally does this configuration in a different way - If libpq and the stdlib ssl module are both linked against the *same* copy of openssl, then they can end up fighting with each other - So the psycopg2 code has a special hack to unconditionally disable libpq's thread-safety code, because the psycopg2 developers assumed that psycopg2 and the stdlib ssl would *always* share the same copy of openssl, and the stdlib ssl module would take care of the thread-safety stuff - Then they started distributing psycopg2 binaries with their own copy of openssl in them, and of course they got crashes, because they've turned off thread-safety, and now that they have their own copy of openssl, no-one else is fixing it for them

So all they need to do to fix their wheels is either:

- somehow disable this patch in their wheel builds: https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a8...

Or else:

- switch to building their wheels against a newer version of openssl, since newer versions of openssl are now thread-safe by default (thank goodness)

Either way, it's totally fixable, and only the psycopg2 devs can fix it.

I don't work on psychopg2, nor do I understand the threading issues, but I offered to help, current status here: https://github.com/psycopg/psycopg2-wheels/pull/8 Cheers, Matthew