Wiki Migration Update
Hello,
Once again, I've had some time to push the wiki migration along. As always, the results can be found here:
I've updated the archived content to that of 16th July, so the translated content should reflect the existing wiki fairly accurately.
Users
Big changes this time have happened around the handling of user information. I have modified the MoinMoin package installer to preserve author/editor details, a capability which for some reason had been removed from MoinMoin. I also need to preserve timestamps as well so that the history of each page makes a bit more sense, but at this point you should be able to see individual contributions:
http://mmwiki.boddie.org.uk/DEV/LogoSubmissions?action=info
The XML export archives do not contain user details over and above who edited which page (or who uploaded which attachment), but I have written scripts to obtain usernames from the archives, to fetch user details from the existing wiki, and to create accounts for users in MoinMoin. An alternative might involve getting a database query run on the server to provide user details:
https://confluence.atlassian.com/display/CONFKB/How+to+Export+User+Data+to+C...
There's also some kind of export functionality in Atlassian OnDemand, whatever that is:
https://confluence.atlassian.com/display/AOD/Exporting+wiki+data
But the solution above may be good enough: it just creates an account for each existing user and employs the same username, full name (or alias) and e-mail address, and sets a random password. It also obtains the URL of any profile picture, with these user details obtained by scraping the home page of each user on the existing wiki. How people then log into their accounts is something we can decide: e-mailing the random password is not exactly secure, but they could reset their account in the MoinMoin interface instead (which involves a reset e-mail that isn't too secure either, but it's the slightly better choice).
Redirects
One thing we also discussed were the redirects that occur for "tiny URLs" and where page identifiers are employed. For example:
http://mmwiki.boddie.org.uk/x/AgA3 http://mmwiki.boddie.org.uk/pages/viewpage.action?pageId=3604482
A mapping of page identifiers is extracted from the exported data, and this mapping is used to support a simple redirection script that looks up the appropriate page name and redirects to that page. One thing it does not yet do is to redirect to a specific revision, however.
To Do
I always provide a list of things that still need doing, so here are some familiar items:
The way comments are presented on pages still needs improving. I may write a macro to include the comment pages in a nicer way and maybe even to allow new comments to be added. Meanwhile, the comment pages should now only be editable by their original authors (by applying ACLs).
It might also be nice to have a list of attachments on pages that have them, and I will take a look to see how Confluence tends to present such things.
User home pages should probably be populated and have things like profile images (if provided), activity indicators, and maybe the dashboard functionality should be emulated, too.
In Conclusion
As I have mentioned previously, the source code for the converter can be found here:
http://hgweb.boddie.org.uk/ConfluenceConverter/
Please take a look at your favourite pages and let me know where improvements can be made to the conversion process. I usually apologise for pages with question marks at the end of their names, which is a lot of the FAQ, but I've implemented a mod_rewrite hack that lets you view them, so if your passion is for FAQ fidelity then please take this opportunity to have a look.
I hope this is of interest!
Paul
On Jul 19, 2013, at 01:15 AM, Paul Boddie wrote:
Once again, I've had some time to push the wiki migration along. As always, the results can be found here:
I've updated the archived content to that of 16th July, so the translated content should reflect the existing wiki fairly accurately.
Thanks very much for continuing to work on this Paul. I think it's looking great and I'd love to start coming up with a plan to migrate officially. Input from top wiki editors is key, so perhaps Mark, Terri, Steve and others can chime in. I can tell you though that my motivation to garden the current wiki is pretty low, and I think I'd be more motivated to clean it up after the Moin migration. If the other wiki editors feel the same way, then we should seize the opportunity to get the migration started.
What features are missing from Moin that would prevent us from migrating, or make it more painful than staying with Confluence for now?
But the solution above may be good enough: it just creates an account for each existing user and employs the same username, full name (or alias) and e-mail address, and sets a random password. It also obtains the URL of any profile picture, with these user details obtained by scraping the home page of each user on the existing wiki. How people then log into their accounts is something we can decide: e-mailing the random password is not exactly secure, but they could reset their account in the MoinMoin interface instead (which involves a reset e-mail that isn't too secure either, but it's the slightly better choice).
I'm happy with just asking everyone to reset their password. We can't do it on the demo site because email is disabled currently.
What can we do about setting up the ACLs for editing? I'm happy if we start by enabling a small group to start with and having folks who want to edit pages re-request enabling their editing permissions.
One thing we also discussed were the redirects that occur for "tiny URLs" and where page identifiers are employed. For example:
http://mmwiki.boddie.org.uk/x/AgA3 http://mmwiki.boddie.org.uk/pages/viewpage.action?pageId=3604482
A mapping of page identifiers is extracted from the exported data, and this mapping is used to support a simple redirection script that looks up the appropriate page name and redirects to that page. One thing it does not yet do is to redirect to a specific revision, however.
I think that's fine. I don't mind if redirects to anything other than the latest revision is enabled. Mark might disagree.
I guess we won't be able to use tiny urls after the migration though, right?
I always provide a list of things that still need doing, so here are some familiar items:
The way comments are presented on pages still needs improving. I may write a macro to include the comment pages in a nicer way and maybe even to allow new comments to be added. Meanwhile, the comment pages should now only be editable by their original authors (by applying ACLs).
It might also be nice to have a list of attachments on pages that have them, and I will take a look to see how Confluence tends to present such things.
User home pages should probably be populated and have things like profile images (if provided), activity indicators, and maybe the dashboard functionality should be emulated, too.
Would this work be better tackled before an official migration or can some of these things be done after the fact?
I hope this is of interest!
It *is*, very much so. Thanks again!
-Barry
On 07/26/2013 12:59 PM, Barry Warsaw wrote:
On Jul 19, 2013, at 01:15 AM, Paul Boddie wrote:
Once again, I've had some time to push the wiki migration along. As always, the results can be found here:
I've updated the archived content to that of 16th July, so the translated content should reflect the existing wiki fairly accurately.
Thanks very much for continuing to work on this Paul. I think it's looking great and I'd love to start coming up with a plan to migrate officially. Input from top wiki editors is key, so perhaps Mark, Terri, Steve and others can chime in.
Yes, thank you Paul. I too think it looks really good.
I can tell you though that my motivation to garden the current wiki is pretty low, and I think I'd be more motivated to clean it up after the Moin migration. If the other wiki editors feel the same way, then we should seize the opportunity to get the migration started.
+1
What features are missing from Moin that would prevent us from migrating, or make it more painful than staying with Confluence for now?
I think we can do alright with a switch. I support a Moin wiki for my bike club, and I think we'll be OK. I have a bit to add below.
But the solution above may be good enough: it just creates an account for each existing user and employs the same username, full name (or alias) and e-mail address, and sets a random password. It also obtains the URL of any profile picture, with these user details obtained by scraping the home page of each user on the existing wiki. How people then log into their accounts is something we can decide: e-mailing the random password is not exactly secure, but they could reset their account in the MoinMoin interface instead (which involves a reset e-mail that isn't too secure either, but it's the slightly better choice).
The reset email shouldn't be a big issue if the user is prepared to receive and act on it. The reset token is only valid during the window from when it is sent to when it is used.
I'm happy with just asking everyone to reset their password. We can't do it on the demo site because email is disabled currently.
What can we do about setting up the ACLs for editing? I'm happy if we start by enabling a small group to start with and having folks who want to edit pages re-request enabling their editing permissions.
It's fairly simple to add ACLs to pages that give one or more groups various access rights. Groups are just wiki pages with a list of the login names of the members of that group and so are administered by the members of the group that can modify that page.
I think we can deal with that by making a small 'admin' group and then having people at large request permission (i.e. addition to the larger 'authors' group as they do now.
There is an issue on my own Moin wiki in that people who are logged in but not members of any group can create and edit pages which don't have ACLs. At one point I was discovered by wiki spammers. I 'fixed' that by using Moin's textcha facility to effectively require a password to create an account. It could be solved more readily by not giving 'Known' users default (or any) write access.
One thing we also discussed were the redirects that occur for "tiny URLs" and where page identifiers are employed. For example:
http://mmwiki.boddie.org.uk/x/AgA3 http://mmwiki.boddie.org.uk/pages/viewpage.action?pageId=3604482
A mapping of page identifiers is extracted from the exported data, and this mapping is used to support a simple redirection script that looks up the appropriate page name and redirects to that page. One thing it does not yet do is to redirect to a specific revision, however.
I think that's fine. I don't mind if redirects to anything other than the latest revision is enabled. Mark might disagree.
I do not disagree. I see no need to redirect those URLs to other than the current page.
I guess we won't be able to use tiny urls after the migration though, right?
I think that's right, and it's unfortunate, but I can live with it (or implement a tiny url feature for Moin).
I always provide a list of things that still need doing, so here are some familiar items:
The way comments are presented on pages still needs improving. I may write a macro to include the comment pages in a nicer way and maybe even to allow new comments to be added. Meanwhile, the comment pages should now only be editable by their original authors (by applying ACLs).
It might also be nice to have a list of attachments on pages that have them, and I will take a look to see how Confluence tends to present such things.
User home pages should probably be populated and have things like profile images (if provided), activity indicators, and maybe the dashboard functionality should be emulated, too.
Would this work be better tackled before an official migration or can some of these things be done after the fact?
I think after the fact, at least for the home page stuff, is fine.
I hope this is of interest!
It *is*, very much so. Thanks again!
Yes, absolutely +1
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Jul 26, 2013, at 02:05 PM, Mark Sapiro wrote:
I can tell you though that my motivation to garden the current wiki is pretty low, and I think I'd be more motivated to clean it up after the Moin migration. If the other wiki editors feel the same way, then we should seize the opportunity to get the migration started.
+1
That's encouraging! :)
I think the general plan is to host the wiki on the python.org infrastructure. They're already hosting the Python and Jython wikis and Noah seemed amenable to the idea at the last Pycon.
If there are no objections, I'll reach out to the infrastructure team to see what they'd need. I'll have to contact John and Matt to do any required DNS changes. I think the rough task list would be:
- Get python.org to assign us an IP address
- Map wiki-new.list.org to that IP
- Freeze all edits on wiki.list.org
- Have Paul do one more migration that he's happy with
- Get pdo to install that on wiki-new
- Test, test, test
- Move the DNS for wiki.list.org to the new IP
- Decommission the Confluence instance
I think we can do alright with a switch. I support a Moin wiki for my bike club, and I think we'll be OK. I have a bit to add below.
Would you and/or Paul want shell access to the new Moin instance? I don't know if that's possible, but if you do, I'd make that part of my request to infrastructure.
There is an issue on my own Moin wiki in that people who are logged in but not members of any group can create and edit pages which don't have ACLs. At one point I was discovered by wiki spammers. I 'fixed' that by using Moin's textcha facility to effectively require a password to create an account. It could be solved more readily by not giving 'Known' users default (or any) write access.
Is that possible? I think we'd definitely want to do that. Also, I guess 'unknown' users would also not have any write access, correct?
I guess we won't be able to use tiny urls after the migration though, right?
I think that's right, and it's unfortunate, but I can live with it (or implement a tiny url feature for Moin).
Yay for open source! :) I love the tiny urls. I think it would make a great addition to Moin in general.
I think after the fact, at least for the home page stuff, is fine.
Cool.
-Barry
On 07/26/2013 02:25 PM, Barry Warsaw wrote:
On Jul 26, 2013, at 02:05 PM, Mark Sapiro wrote:
If there are no objections, I'll reach out to the infrastructure team to see what they'd need. I'll have to contact John and Matt to do any required DNS changes. I think the rough task list would be:
- Get python.org to assign us an IP address
- Map wiki-new.list.org to that IP
- Freeze all edits on wiki.list.org
- Have Paul do one more migration that he's happy with
- Get pdo to install that on wiki-new
- Test, test, test
- Move the DNS for wiki.list.org to the new IP
- Decommission the Confluence instance
Sounds good. Re: timing, I am off line much of now through Aug 30, so for me at least, 'Test, test, test' shouldn't begin before September.
Would you and/or Paul want shell access to the new Moin instance? I don't know if that's possible, but if you do, I'd make that part of my request to infrastructure.
It might be helpful, but the more critical things are that probably at least one of us should be in the superuser list (a configuration setting) and there should be an AdminGroup with all rights and we should be in it.
Of course we can do this with shell access to the configuration file.
There is an issue on my own Moin wiki in that people who are logged in but not members of any group can create and edit pages which don't have ACLs. At one point I was discovered by wiki spammers. I 'fixed' that by using Moin's textcha facility to effectively require a password to create an account. It could be solved more readily by not giving 'Known' users default (or any) write access.
Is that possible? I think we'd definitely want to do that. Also, I guess 'unknown' users would also not have any write access, correct?
My Moin installation is dumb in this respect. I don't remember why we wanted to set it up the way we did, maybe so a user could create their own home page before being added to a group, but in this age of wiki spam it's probably not a good idea.
And yes, we can be as loose or as tight as we want. The page at https://moinmo.in/HelpOnAccessControlLists gives the story, but the short answer is we can require a user to be logged in and a member of some specific group before they can create or modify any content.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Friday 26. July 2013 21.59.14 Barry Warsaw wrote:
[Stuff already discussed]
What features are missing from Moin that would prevent us from migrating, or make it more painful than staying with Confluence for now?
I think only the actual wiki editors can answer this. I imagine that the dashboard features would be nice to have.
[...]
I'm happy with just asking everyone to reset their password. We can't do it on the demo site because email is disabled currently.
I think e-mail is enabled on the python.org system, so they should be able to reset their passwords.
[...]
I always provide a list of things that still need doing, so here are some familiar items:
The way comments are presented on pages still needs improving. I may write a macro to include the comment pages in a nicer way and maybe even to allow new comments to be added. Meanwhile, the comment pages should now only be editable by their original authors (by applying ACLs).
I've now made some support for comments, including adding new ones. It might be wise to add textcha integration for this, though, but I already support ACLs and thus don't allow the posting of comments on pages where a user cannot write to that page, even though the comments are actually subpages.
It might also be nice to have a list of attachments on pages that have them, and I will take a look to see how Confluence tends to present such things.
User home pages should probably be populated and have things like profile images (if provided), activity indicators, and maybe the dashboard functionality should be emulated, too.
Would this work be better tackled before an official migration or can some of these things be done after the fact?
I don't think making user pages would be too difficult, but populating them with fancy features would probably end up happening later.
Paul
On Friday 26. July 2013 23.25.12 Barry Warsaw wrote:
On Jul 26, 2013, at 02:05 PM, Mark Sapiro wrote:
I think the general plan is to host the wiki on the python.org infrastructure. They're already hosting the Python and Jython wikis and Noah seemed amenable to the idea at the last Pycon.
If there are no objections, I'll reach out to the infrastructure team to see what they'd need. I'll have to contact John and Matt to do any required DNS changes. I think the rough task list would be:
- Get python.org to assign us an IP address
- Map wiki-new.list.org to that IP
- Freeze all edits on wiki.list.org
- Have Paul do one more migration that he's happy with
- Get pdo to install that on wiki-new
- Test, test, test
- Move the DNS for wiki.list.org to the new IP
- Decommission the Confluence instance
We can iterate on the migration a few times before freezing and doing the final migration. The testing part is the important part, of course. ;-)
I think we can do alright with a switch. I support a Moin wiki for my bike club, and I think we'll be OK. I have a bit to add below.
Would you and/or Paul want shell access to the new Moin instance? I don't know if that's possible, but if you do, I'd make that part of my request to infrastructure.
It would probably help a lot.
There is an issue on my own Moin wiki in that people who are logged in but not members of any group can create and edit pages which don't have ACLs. At one point I was discovered by wiki spammers. I 'fixed' that by using Moin's textcha facility to effectively require a password to create an account. It could be solved more readily by not giving 'Known' users default (or any) write access.
Is that possible? I think we'd definitely want to do that. Also, I guess 'unknown' users would also not have any write access, correct?
We can define a group of trusted editors and give them full rights. For everyone else, there is a range of options including making them ask for permission on the mailing list (or mailing an administrator directly) before being able to write anything, making them log in and possibly having "textcha" (challenge question) verification of their edits, or just having textchas preventing people from spamming.
(Textchas don't work well on the Python Wiki at the moment because the questions seem to be easily guessed and haven't been fixed to address this, as far as I can tell.)
I see that Mark has probably summarised this better than I have. :-)
I guess we won't be able to use tiny urls after the migration though, right?
I think that's right, and it's unfortunate, but I can live with it (or implement a tiny url feature for Moin).
Yay for open source! :) I love the tiny urls. I think it would make a great addition to Moin in general.
Some kind of "compression" of a page name might be enough to provide a similar feature without requiring some kind of special index, but something using an index could also be done if necessary.
Paul
On Friday 26. July 2013 23.55.00 Mark Sapiro wrote:
Sounds good. Re: timing, I am off line much of now through Aug 30, so for me at least, 'Test, test, test' shouldn't begin before September.
Leaving any serious activity for a few weeks would suit me, too.
[...]
My Moin installation is dumb in this respect. I don't remember why we wanted to set it up the way we did, maybe so a user could create their own home page before being added to a group, but in this age of wiki spam it's probably not a good idea.
On the subject of home pages, users can create their own after creating an account, but it is also possible to impose textchas on the registration process itself. The global ACLs may also prevent the creation of home pages through things like the MyPages action, but I haven't really checked this.
Paul
On 07/26/2013 03:59 PM, Paul Boddie wrote:
On the subject of home pages, users can create their own after creating an account, but it is also possible to impose textchas on the registration process itself.
That is in fact what I do. I have one TextCha question, "What is the account creator password?" and the answer is given to people out of band.
I also have a SuperGroup whose members are all the other groups, and (indirect) members of the SuperGroup are exempt from TextChas.
This way, one needs to establish contact before registering, and once registered and added to at least one group, they can proceed with minimal hinderance.
The global ACLs may also prevent the creation of home pages through things like the MyPages action, but I haven't really checked this.
I'm pretty sure this is the case.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 07/26/2013 03:38 PM, Paul Boddie wrote:
On Friday 26. July 2013 21.59.14 Barry Warsaw wrote:
What features are missing from Moin that would prevent us from migrating, or make it more painful than staying with Confluence for now?
I think only the actual wiki editors can answer this. I imagine that the dashboard features would be nice to have.
I'm only one data point, but pretty much the only thing I use the dashboard for is seeing the recently updated pages, both to see what's going on and check for spam. I'm perfectly happy with Moin's RecentChanges page for this.
[...]
I've now made some support for comments, including adding new ones. It might be wise to add textcha integration for this, though, but I already support ACLs and thus don't allow the posting of comments on pages where a user cannot write to that page, even though the comments are actually subpages.
Cool on the comments support.
On the ACLs, are you assuming acl_hierarchic is True or are you just duplicating the parent page's ACL?
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Jul 26, 2013, at 02:55 PM, Mark Sapiro wrote:
Sounds good. Re: timing, I am off line much of now through Aug 30, so for me at least, 'Test, test, test' shouldn't begin before September.
Works for me too. What I'll do in the meantime is get the ball rolling with infrastructure, try to get an IP assigned, and get the wiki-new.list.org hostname assigned to it. We can do all the rest later.
-Barry
On Saturday 27. July 2013 02.23.41 Mark Sapiro wrote:
I'm only one data point, but pretty much the only thing I use the dashboard for is seeing the recently updated pages, both to see what's going on and check for spam. I'm perfectly happy with Moin's RecentChanges page for this.
Right. It might be nice to have a per-user RecentChanges macro showing only a particular user's edits, and I've been meaning to look into this for a while, anyway.
[...]
Cool on the comments support.
On the ACLs, are you assuming acl_hierarchic is True or are you just duplicating the parent page's ACL?
I actually set an ACL that lets only the author of the comment edit the comment subpage. This obviously doesn't let others add replies to an individual comment, however, so maybe a more sophisticated approach is necessary.
Paul
On 07/26/2013 06:23 PM, Mark Sapiro wrote:
I'm only one data point, but pretty much the only thing I use the dashboard for is seeing the recently updated pages, both to see what's going on and check for spam. I'm perfectly happy with Moin's RecentChanges page for this.
I guess I can be data point number 2: I also mostly use the dashboard for seeing recent stuff, either what's going on or spam, so RecentChanges will also work for me too.
Terri
participants (4)
-
Barry Warsaw -
Mark Sapiro -
Paul Boddie -
Terri Oda