[Chicago] AppEngine Bulk Loading

[Kumar McMillan]

On Mon, May 12, 2008 at 8:29 PM, Feihong Hsu <hsu.feihong at yahoo.com> wrote:
> Argh. I can't use "line.encode('ascii', 'replace')" because many of
> the strings in my app are ALL Unicode, and there would be no
> distinction between them anymore.
>
> Well, the BulkLoad class is just another WSGI app, so I can write my
> own custom bulk loading app and it shouldn't perform any worse,
> right?

you can copy it out and patch it up for some Unicode stuff but you
might still run in to trouble with the internal datastore Unicode
problems.  I don't know how far you can get patching internal
datastore objects in the production env but it's worth a try ;)

>
> Anyway, I have about 45k entities to load. Is there anyone with
> experience bulk loading a large number of items to AppEngine? Do I
> even want to know how long that's gonna take?

if you try it on your local dev server then you are going to see
things grind to sad, pathetic halt very soon.  It seems to me that the
SDK version of the datastore is a dict object or something.  I.E. fill
it up with more than 1000 rows and your computer will cry.

That said, you're best off just getting it all to work on your
production db.  It's very fast to bulk load in production.