> For example, matplotlib 1.0 was released in July 2010, 2.0 in January 2017, then 3.0 in Sept 2018

In defense of Matplotlib, we did the 1.0 -> 2.0 change when we changed the default style, which is a poster child for "silent significant changes"! and the 2.0 -> 3.0 transition when we dropped Python 2 support (but had minimal other changes).  I do think we should be bumping major versions more often, but not so that we can break more things, but because platonic semver is effectively a lie and we should be more honest about the deprecations / changes we are making.

I am weakly in favor of the 

import skimage.v0
import skimage.v1 

pattern.  You can put the v0 namespace in _now_ and start documenting it as the preferred way with the try..except ImportError.. pattern to allow back-compat support with all currently released versions.  This is also a mechanical enough of a transformation that a futurize style translator seems plausible (I think "sorry, we want you to change all your imports...but we have a script to do it for you!" will buy a lot of user goodwill).  You might even want to in a 0.x release start decorating all of the functions accessed through the top-level namespace (not the vX namespace) to suggest that the imports need to be fixed or versions pinned (this would be some slightly garnly meta-programming but I think doable with decorators + module properties).

As you start working on 1.0, anything in v0 that changes turns into a wrapper that reproduces the old API (which there are going to be some users that really want to keep the old API so keeping the v0 shims in skimage and testing them helps ensure there really is a path back!).  When you release v1, you freeze, the v0 API (no new features, only super critical bug fixes (~security or segfault), if functions break due to upstream changes they get deprecated and removed) and let is live ~forever (code you do not worry about is not a huge maintenance burden) and leave the top-level as-is (or add/ramp up the warnings on how to import v0 _or_ migrate to v1), and in v1.3 or something like that drop the top-level access so users always do

import skimage.v1 as skimage

At NSLS-II we have gone through some major API and implementation re-thinks on one of our projects, used a version of the above scheme and have been pretty happy.  Having the ability to get both the "old" and the "new" in the same Python process is invaluable as it lets you operate in a mixed mode where you have _some_ code in the notebook/script which is still using the v0 API but start writing new code in the same namespace that uses the new API.  This is also valuable for down-stream libraries as it could let the developers migrate gradually to the new API without the need for a "flag day" change.  This scheme also side-steps the "diamond dependency" issue as both APIs will be supported!

Put another way, you do not want to put a graduate student in the position of saying "I _want_ to use the new API, but I have 10k LoC of inherited code using the old API .....". 

I think doing it within the same top-level package is better than making a new top-level package because it is clear it is the _same_ project not a (hostile) fork or competitor.

I'm with Stéfan that subtle version shenanigans will do more harm than good.

A very bad idea that this would also allow is to have a helper function like `make_top_level_api('v1')` which would allow you to globally change which version is exposed ;) 

There is some precedent for this in libhdf5 who when they break API add a new (numbered) function and then provide c macros to let you pick a coherent set of them with un-numbered names (or mix-and-match with the numbered versions if you want).

At the end of the day, this change is going to cause (some) pain to someone (everyone?). The more you can make the pain for the users scale with their benefit/buyin the better!  


On Thu, Jul 22, 2021 at 4:37 AM Juan Nunez-Iglesias <jni@fastmail.com> wrote:
I think Nelle is concerned that doing it once (and surviving 😉) is enough to trigger a mindset change in the core developers that breaking releases are Ok, and thus can be done more frequently. For example, matplotlib 1.0 was released in July 2010, 2.0 in January 2017, then 3.0 in Sept 2018. 😬

In private discussions with Nelle she suggested that she *also* didn’t like skimage2, and in fact would advocate for gradual but breaking changes — meaning, at some point, `filters.gaussian(image)` becomes an *error* without `filters.gaussian(image, preserve_range=True)`. Then in later versions we deprecate that arg. This can all happen very slowly but the key is that the silent behaviour change is extremely noisy somewhere in the middle. You end up with a silent behaviour change but not before everything breaks loudly for an as-yet-unspecified period of time. The key is that people pay attention to errors, but tend to ignore warnings.

Now, for some, that *final* state, that 0.19 code ends up working differently in 0.23, even though it was broken in 0.22, is enough to say, let’s go with new-name/new-import. That’s fine.

But for others, one big issue with 1.0 is that it was impossible to warn loudly enough. So I’d like to propose what I call the chaotic good modification to the SKIP as an option:

- 0.19 is released, no deprecations, no nothing, everything is as it always was.
- over six months, we break everything we want to break on master on the road to the brave new 1.0 world.
- once we have everything where we want it, we release 0.20, identical to 0.19 but with a warning to migrate. We simultaneously release 1.0b0, so that people can migrate and depend explicitly on scikit-image>=1.0b0.
- after six months, we release 0.21. This is a completely broken release that does not import: it raises an exception that you *must* either migrate or pin to continue using scikit-image.
- after an unspecified period of time, we finally release 1.0.

Releasing a broken package is definitely chaotic, but it *does* force people to pay attention, and if the error makes the fix obvious enough, people are ok with this.

To me, this really dramatically diminishes the risk that the transition will go unnoticed by most people.

The major downside relative to the new package/new import solution is still that you *do* make 10y of stack overflow answers obsolete in some form or another. And, to be honest, many people don’t install from requirements files, they install with `pip install [package that I am reading about for the first time]`.

Anyway, I think that in a perfect world, Riadh, yes, the version bump should be enough signal. But it’s not a perfect world and users have expectations. I don’t think it’s good enough from a “customer satisfaction” standpoint if we just say “sorry that you didn’t read the fine print on the label and are now missing an eyeball” — we should indeed try to put more measures in place than a small label to prevent people from poking their eyeballs out. 😜

As a real-world example, a bunch of people in Australia got burnt by their thermomixes when they sloshed too much hot liquid around, even though the instructions say don’t use turbo mode on hot liquids. Thermomix ended up shipping TM5s (new model with more safety features) to everyone who purchased a TM31 in the last x months of its life. (Including yours truly) Moral of the story: let’s try from the beginning not to burn our users even if they didn’t read the label. 😉

Although, the counterpoint: it is super annoying that the TM5 holds the lid closed for 8 seconds now before letting you open it. 😂


On 21 Jul 2021, at 7:06 pm, Riadh <rfezzani@gmail.com> wrote:

Le 21/07/2021 à 10:55, Nelle Varoquaux a écrit :

I would be a bit cautious a bit about the "changes are not silent because the major version number is upgraded." That opens the door to doing a lot more major versions in order to "allow" for API breakage when it could be avoided.

As a user, I find that it would be nice if my code that only depends on numpy, scipy, and matplotlib that I started at the beginning of a research project with up-to-date packages also worked at submission time with up-to-date versions of those packages with minimal changes to the code :p


That's not what I am calling for, we are talking here about a first major release after a years long development  period.

I am absolutely not pushing for a faster release pace ;)


scikit-image mailing list -- scikit-image@python.org
To unsubscribe send an email to scikit-image-leave@python.org
Member address: jni@fastmail.com

scikit-image mailing list -- scikit-image@python.org
To unsubscribe send an email to scikit-image-leave@python.org
Member address: tcaswell@gmail.com

Thomas Caswell