I hope Abhilash doesn't mind my following up to the mailing list. This is useful information that should be searchable in the archives.

Abhilash is working on a port of the core from Storm to SQLAlchemy as the Python ORM layer. As an aside, this port plus the one I have pending for replacing restish with falcon, would allow us to start the port of Mailman 3 to Python 3.

On Sep 07, 2014, at 06:15 AM, Abhilash Raj wrote:

I am facing a problem where I get this error:

mailman.interfaces.domain.BadDomainSpecificationError: Duplicate email host: example.com

while running the tests, now I suspect this probably because the database is not being reset in between tests.

This is exactly right. Here's how the test suite ensures a clean system for each test. Note that this includes resetting the database, but also taking care of things like clearing out the queue directories and other persistent state.

The first place to look is in the test layers. These are a Zope-ism adopted by nose2 (the test runner that MM3 uses) that provides for fixture-like resource management.

https://nose2.readthedocs.org/en/latest/plugins/layers.html

MM3's layers are defined in src/mailman/testing/layers.py. The most common layer is ConfigLayer which ensures that the configuration subsystem is initialized for the tests. ConfigLayer's testTearDown() is run at the end of every test that uses the ConfigLayer or any of its subclass layers. This method is pretty simple; it just calls reset_the_world() which is where all the good stuff happens. See src/mailman/testing/helpers.py.

Most of that method should be pretty obvious; the call to config.db._reset() is where the database gets cleared out. _reset() is actually defined in src/mailman/database/factory.,py and placed onto the database object in DatabaseTestingFactory.create(). So how does _reset() work?

First, it rolls back the database just to throw away any in progress transactions. It then calls back into the specific database implementations to give them a chance to do some pre-rest actions. It'll do something similar after the next step - it'll make a post-reset callback and then commit any outstanding transactions. The real heart of the method is in the call to ModelMeta._reset()[1].

ModelMeta is the metaclass for all ORM classes, via the Model base class used everywhere. The idea here is that each ORM class registers itself in ModelMeta._class_registry(), as long as the ORM class doesn't have a PRESERVE=False flag, indicating that particular table should *not* be reset after every test (currently only the Version table has this setting).

Now you can see what ModelMeta._reset() does. It iterates through every registered ORM class, and removes every row in the associated database. I do it this way instead of dropping every table because it's actually more difficult to re-establish the schema than it is to remove the rows.

So the trick for the SQLAlchemy port is to figure out both how to identify which tables to clear out, and how to drop all of those table's rows.

Hope that makes sense!

Cheers, -Barry

[1] I just noticed that ModelMeta._reset() calls _pre_reset() again. That's probably a lurking bug that should be fixed.