On Sun, Nov 12, 2017, at 08:06 AM, Simon Hanna wrote:
These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :) This contradicts what you said in my merge request for Postorius https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756
You either use released versions, or make sure that the master branches are always compatible. You can't have both with the current development model. Not exactly, I still believe all our git-heads should be compatible with released (i.e. stable) versions of dependencies, regardless of the fact that we maintain those dependencies.
Also, there is at least one test per-project which tests with git-heads of dependencies we maintain, so that we know if we make any incompatible changes.
I am not sure what in the current development model prevents us to do both?
The story for container images is different than testing with git-heads. We need better integration testing for the entire suite to be stable in conjunction, rather than independently. In fact, containers might be the perfect way to actually do the integration testing. Scenario: Core changes something about rest that is backwards incompatible. The change is commited to master. Since your containers use the master branches, all future builds will fail until mailmanclient and postorius/hyperkitty.
Yes, that is a totally possible scenario.
But, given that kind of testing we currently do, this scenario can happen not only to git-heads but also releases. That is what I am trying to build using container images.
The fact that I *announced* and *released* these rolling images for public use, is only because people asked for it before. These images come with a fair amount of warning that, right now, they can break!
If you want to always have Postorius (and others) compatible with the latest release of master, development for Postorius will be a Pain, because you can't use the master branch of mailman for testing and quick changes. Also your containers and all integration tests that can be done with them will always fail.
It would most definitely be a pain if we were to break REST API frequently, but that is quite rare. Given that in future API will be consumed by not just us, it is important to keep this assumption true.
Another wishful project I have is to be able to "map" the REST API and monitor for changes; it'd also help us with lemme (the authenticating/authorization proxy for Core).
And yes, container images will fail, and that's the point! I want them to fail so that releases don't :)
Rolling images are meant to bring a faster feedback-loop from users who are OK with rare occasional breaking and want to help test the software, something like public-beta releases.
Until you fix Postorius and the other clients... So why not just have the master branches always compatible and make life easier for developers.
Incompatible changes are released only with major version upgrades, so it is not going to happen with everyone, unless they choose to use the git-heads and get themselves into it.
I don't see a good reason why you want to do it the other way round...
Any bug in core that affects Postorius will also just remain there until you release a new version of core.
If you really really want to have it your way, there have to be more frequent releases.
Yes, we definitely don't want to say "use new beta-quality container images or stone age stable release". Going forward, we will have more frequent releases too :)
- Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services. The current bottleneck is the rest api from core. IMO bulk actions, filtering and sorting should be added there before trying to have multiple postorius/hyperkitty instances serving the same core... I didn't say they will be served from the same core ;-)
Yes, the Core's API can be further improved to be more efficient. But that doesn't prevent us from scaling. I understand there are issues around multiple instances of containers, but I have ideas to get around most of them. I just don't want that people that have issues with big installations get the answer: We know of that issue, please deploy Mailman on X containers/machines as a workaround
Containers are just an easy way to scale, given that we already have them, it'd be fun to see if I can make multiple instances run in parallel. Given that _most_ parts store their data in a data store (database, redis etc), it shouldn't be too difficult. There will be quirks with logging, queues in Core and hence held message views in Postorius, but apart from that, I think everything else can be served using any random instance of Core/Postorius/Hyperkitty.
We can most definitely run multiple instances of Hyperkitty, which I feel will be more used than any other part of UI. If we partition the data smartly, we could do the same with Core.
Again, these are just theories and needs much more testing and some changes in components themselves! This would most definitely increase the availability and auto-scaling.
Making Core's API more efficient would definitely be a good thing and we do intend to do that in future. But, this would still be relevant if we had an efficient REST API, it's not a band-aid to the actual problem :)
-- Abhilash Raj maxking@asynchronous.in