[Tracker-discuss] Working with the sourceforge screenscraper

["Martin v. Löwis"]

Erik Forsberg schrieb:
> On the other hand, using real data as often as possible makes sure we
> find as many importer bugs as possible, and also forces us to keep the
> importer up to date with any changes (layout-wise etc.) at sourceforge
> so that we don't find out in the very last moment that we can't import
> data. 

I also think we should have real data online before the actual switch
is made. People should be able to look at it and make comments before
it goes life. The length of such a "sunrise" period is debatable; I
think two weeks should be sufficient (or even less).

Keeping these data up-to-date in an ongoing manner is not important,
IMO.

> Just for your knowledge, "a single run of the transfer script" takes
> many hours and is, due to the instability of sourceforge, a very
> error-prone project. Not to mention that sourceforge block me for half
> an hour once in a while for fetching too much data. 

Yes, we can't do that too often (and yes, I understand it is tedious
for whoever is doing it). If it means that the tracker is frozen for
two or three days: that's fine. If then something is lost because
people were posting messages even though it was frozen: that's fine
as well.

> I would be much more comfortable if we had a process where step 1) is
> always kept in sync with sourceforge. This would keep us much less
> dependent on sourceforge being up and working at the time for the
> final conversion.

Don't worry about it too much. If we can blame SF, that's fine. People
understand that such problems are the primary rationale for this project
in the first place.

If you find after day one of the conversion that SF just won't give
you the data, we lift the freeze and wait a week to try again.

Regards,
Martin