Using hyperkitty_import for archives downloaded from HK
I noticed that hyperkitty_import has issues with In-Reply-To header when importing archives downloaded from Hyperkitty itself.
In some cases in_reply_to field in hyperkitty_email table will have extra white space at the start. For example In-Reply-To: CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com will be saved as ' CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ'.
This results in broken threads, as parent of this message cannot be found.
I also noticed that hyperkitty_import doesn't have this problem with archives from MM2, but in MM2 archives, message id in in-reply-to header is surrounded with brackets, like this: In-Reply-To: <CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com>
When I look at original email source, in-reply-to header doesn't have brackets; it looks like in the archive made by HK.
Is the issue here with import or export (download) process? Is hyperkitty_import right tool for migrating lists from one MM3 server to another?
I was thinking should I post this to the mm3-users list or here, I hope I made a right decision.
Regards, Goran T.
On Mon, Sep 7, 2020, at 2:25 PM, goran.terzic@gmail.com wrote:
I noticed that hyperkitty_import has issues with In-Reply-To header when importing archives downloaded from Hyperkitty itself.
In some cases in_reply_to field in hyperkitty_email table will have extra white space at the start. For example In-Reply-To: CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com will be saved as ' CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ'.
This results in broken threads, as parent of this message cannot be found.
I also noticed that hyperkitty_import doesn't have this problem with archives from MM2, but in MM2 archives, message id in in-reply-to header is surrounded with brackets, like this: In-Reply-To: <CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com>
When I look at original email source, in-reply-to header doesn't have brackets; it looks like in the archive made by HK.
Is the issue here with import or export (download) process? Is hyperkitty_import right tool for migrating lists from one MM3 server to another?
The import process should be fine, the issues are more likely with download/export process.
Internally, Hyperkitty doesn't store the original email objects, only some of the headers and body of the message. It then generates the export archive on-demand by creating the Email object and stitching them together for mbox archive.
You can just take a database dump and migrate that. If you are using postgres, there is a pg_dump command that I know of. I am sure something exists for Mysql/mariadb too. I would recommend using this method for migrations.
You can also use django's dumpdata1 and loaddata commands to export and import data from one installation to other. It should work when moving from one database type to other in theory. I haven't tested it to comment on how successful it would be.
I was thinking should I post this to the mm3-users list or here, I hope I made a right decision.
Regards, Goran T.
Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-leave@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3
Security Policy: https://wiki.list.org/x/QIA9
-- thanks, Abhilash Raj (maxking)
On Mon, Sep 7, 2020, at 2:25 PM, goran.terzic@gmail.com wrote:
I noticed that hyperkitty_import has issues with In-Reply-To header when importing archives downloaded from Hyperkitty itself.
In some cases in_reply_to field in hyperkitty_email table will have extra white space at the start. For example In-Reply-To: CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com will be saved as ' CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ'.
It turns out HK strips the angle brackets < > when internally storing the message-id and in-reply-to header. It adds them back to Message-ID when exporting but doesn't add it back to In-Reply-To. I have opened 1 for the issue.
Because of that, when the header is folded, an additional whitespace is added to the start resulting in broken threads.
This results in broken threads, as parent of this message cannot be found.
I also noticed that hyperkitty_import doesn't have this problem with archives from MM2, but in MM2 archives, message id in in-reply-to header is surrounded with brackets, like this: In-Reply-To: <CAFV2YWOfZRHEDTTz3F6Emnwx8Sh4_Ar7+JMR7XvOAD8_eOjqaQ@mail.gmail.com>
When I look at original email source, in-reply-to header doesn't have brackets; it looks like in the archive made by HK.
Is the issue here with import or export (download) process? Is hyperkitty_import right tool for migrating lists from one MM3 server to another?
I was thinking should I post this to the mm3-users list or here, I hope I made a right decision.
Regards, Goran T.
Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-leave@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3
Security Policy: https://wiki.list.org/x/QIA9
-- thanks, Abhilash Raj (maxking)
Thanks Abhilash for your quick response
You can just take a database dump and migrate that. If you are using postgres, there is a pg_dump command that I know of. I am sure something exists for Mysql/mariadb too. I would recommend using this method for migrations.
You can also use django's dumpdata1 and loaddata commands to export and
import data from one installation to other. It should work when moving from one database type to other in theory. I haven't tested it to comment on how successful it would be.
Using pg_dump would work when migrating all lists from one server to another, but can't be used for migration of a single list. However, I guess it would be possible to write SQL script for that.
It turns out HK strips the angle brackets
< >when internally storing the message-id and in-reply-to header. It adds them back to Message-ID when exporting but doesn't add it back to In-Reply-To. I have opened 1 for the issue.Because of that, when the header is folded, an additional whitespace is added to the start resulting in broken threads.
Thanks for confirming this issue.
Regards, Goran T
On Mon, Sep 7, 2020, at 4:26 PM, Goran Terzic wrote:
Thanks Abhilash for your quick response
You can just take a database dump and migrate that. If you are using postgres, there is a pg_dump command that I know of. I am sure something exists for Mysql/mariadb too. I would recommend using this method for migrations. You can also use django's dumpdata1 and loaddata commands to export and import data from one installation to other. It should work when moving from one database type to other in theory. I haven't tested it to comment on how successful it would be.
Using pg_dump would work when migrating all lists from one server to another, but can't be used for migration of a single list. However, I guess it would be possible to write SQL script for that.
It turns out HK strips the angle brackets
< >when internally storing the message-id and in-reply-to header. It adds them back to Message-ID when exporting but doesn't add it back to In-Reply-To. I have opened 1 for the issue.Because of that, when the header is folded, an additional whitespace is added to the start resulting in broken threads.
Thanks for confirming this issue.
I also opened the fix 1 for the issue, you can apply it to the source server and export the archives that will import correctly.
Regards, Goran T
-- thanks, Abhilash Raj (maxking)
participants (3)
-
Abhilash Raj -
Goran Terzic -
goran.terzic＠gmail.com