On 07/24/2018 06:32 AM, Andrew Hodgson wrote:

Once done you can see the results by navigating to your Mailman instance and you can see the data about the list such as the creation date, number of posts and the members. For some reason on my instance the last post date is still the last post from Mailman2, even though it has been running on Mailman3 for a month or so.

This is <https://gitlab.com/mailman/mailman/issues/453>, fixed in Mailman 3.2.

Import archives: Use the command python manage.py hyperkitty_import to get the private mbox into your list. I ran into 2 issues:

• If you have posts in the new archive on Mailman3 then you need to specify the earliest date you want to import into the archive as the command won't import anything older than the earliest date in the new archive. If you run the command with the help option you can see the format required;

Yes. I've thought about adding a --all option to import the entire mbox regardless of date, but for now I just specify --since='1990-01-01'.

• There were older messages in the mbox without message-ids in the archive that failed to import. I took the easy way out on this one and didn't import them. In an archive with around 120,000 messages it rejected around 30 messages in all.

This is <https://gitlab.com/mailman/hyperkitty/issues/180>, fixed in HyperKitty 1.2.0.

There are other issues that haven't yet been fixed other than by my own, unmerged patches. See <https://gitlab.com/mailman/hyperkitty/issues/183> and <https://gitlab.com/mailman/hyperkitty/issues/184>.

It really helps to have the import mbox as clean as possible before starting. If it's a MM 2.1 mbox from a list created on MM 2.1.x, it's probably pretty good, but if it's from a list created in MM 2.0 days, it probably has issues. In any case, you need to check it for unescaped 'From ' lines with MM 2.1's bin/cleanarch. Even this isn't perfect, but it helps.

Also, after you import archives, the Haystack search index will not index those messages because they are not seen as new. Older versions of hyperkitty_import would say that the Django minutely update_index job would do it, but it won't. You can run

python manage.py update_index

to index those imported messages, but if you have a large installation with multiple tens of thousands of archived messages, this can take several hours. To help with this, there is now

python manage.py update_index_one_list

to do just a specified list. This too can take a long time (run it in the background) if the imported archive was large, but it's often preferable to doing the entire installation.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 07/25/2018 02:54 AM, Andrew Hodgson wrote:

Thanks for this; I am on the Docker version which is not updated yet. once it gets updated and I migrate to it, will I be able to re-import those missing messages by running a complete import again or will this cause more trouble now?

You can re-run the import with the complete mbox as long as you specify --since= with a date older than any of the messages in the mbox. Without that, it won't import anything because the messages will all be 'too old'.

Running with the full mbox will be OK as those messages with Message-ID: headers will be detected as duplicates and won't be added a second time, but if the mbox is large, it would be faster to run with just the messages without Message-ID: headers, but possibly not faster overall considering the effort to extract those messages.

I did run cleanarch on the mbox before doing any of this import work and didn't get any issues. The lists all came from Mailman 2.1 (starting from around 2.1.5).

That's expected. Since even somewhat before Mailman 2.1.5, archive mboxes have been good. It's normally only those started on Mailman 2.0.x (or older?) or mboxes imported from elsewhere that have significant issues.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan