Using hyperkitty_import for archives downloaded from HK

I noticed that hyperkitty_import has issues with In-Reply-To header when importing archives downloaded from Hyperkitty itself.
In some cases in_reply_to field in hyperkitty_email table will have extra white space at the start. For example In-Reply-To: CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com will be saved as ' CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ'.
This results in broken threads, as parent of this message cannot be found.
I also noticed that hyperkitty_import doesn't have this problem with archives from MM2, but in MM2 archives, message id in in-reply-to header is surrounded with brackets, like this: In-Reply-To: <CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com>
When I look at original email source, in-reply-to header doesn't have brackets; it looks like in the archive made by HK.
Is the issue here with import or export (download) process? Is hyperkitty_import right tool for migrating lists from one MM3 server to another?
I was thinking should I post this to the mm3-users list or here, I hope I made a right decision.
Regards, Goran T.

On Mon, Sep 7, 2020, at 2:25 PM, goran.terzic@gmail.com wrote:
The import process should be fine, the issues are more likely with download/export process.
Internally, Hyperkitty doesn't store the original email objects, only some of the headers and body of the message. It then generates the export archive on-demand by creating the Email object and stitching them together for mbox archive.
You can just take a database dump and migrate that. If you are using postgres, there is a pg_dump command that I know of. I am sure something exists for Mysql/mariadb too. I would recommend using this method for migrations.
You can also use django's dumpdata1 and loaddata commands to export and import data from one installation to other. It should work when moving from one database type to other in theory. I haven't tested it to comment on how successful it would be.
-- thanks, Abhilash Raj (maxking)

On Mon, Sep 7, 2020, at 2:25 PM, goran.terzic@gmail.com wrote:
It turns out HK strips the angle brackets < >
when internally storing the message-id and in-reply-to header. It adds them back to Message-ID when exporting but doesn't add it back to In-Reply-To. I have opened 1 for the issue.
Because of that, when the header is folded, an additional whitespace is added to the start resulting in broken threads.
-- thanks, Abhilash Raj (maxking)

Thanks Abhilash for your quick response
You can also use django's dumpdata1 and loaddata commands to export and
Using pg_dump would work when migrating all lists from one server to another, but can't be used for migration of a single list. However, I guess it would be possible to write SQL script for that.
Thanks for confirming this issue.
Regards, Goran T

On Mon, Sep 7, 2020, at 2:25 PM, goran.terzic@gmail.com wrote:
The import process should be fine, the issues are more likely with download/export process.
Internally, Hyperkitty doesn't store the original email objects, only some of the headers and body of the message. It then generates the export archive on-demand by creating the Email object and stitching them together for mbox archive.
You can just take a database dump and migrate that. If you are using postgres, there is a pg_dump command that I know of. I am sure something exists for Mysql/mariadb too. I would recommend using this method for migrations.
You can also use django's dumpdata1 and loaddata commands to export and import data from one installation to other. It should work when moving from one database type to other in theory. I haven't tested it to comment on how successful it would be.
-- thanks, Abhilash Raj (maxking)

On Mon, Sep 7, 2020, at 2:25 PM, goran.terzic@gmail.com wrote:
It turns out HK strips the angle brackets < >
when internally storing the message-id and in-reply-to header. It adds them back to Message-ID when exporting but doesn't add it back to In-Reply-To. I have opened 1 for the issue.
Because of that, when the header is folded, an additional whitespace is added to the start resulting in broken threads.
-- thanks, Abhilash Raj (maxking)

Thanks Abhilash for your quick response
You can also use django's dumpdata1 and loaddata commands to export and
Using pg_dump would work when migrating all lists from one server to another, but can't be used for migration of a single list. However, I guess it would be possible to write SQL script for that.
Thanks for confirming this issue.
Regards, Goran T
participants (3)
-
Abhilash Raj
-
Goran Terzic
-
goran.terzic@gmail.com