To keep everyone in the loop. I finally had time to sit down and work on this, and I created a patch and submitted a pull request to Martin for his review.<div><br></div><div><a href="https://bitbucket.org/loewis/pypi-appengine/pull-request/1/added-some-logging-to-help-track-mirror">https://bitbucket.org/loewis/pypi-appengine/pull-request/1/added-some-logging-to-help-track-mirror</a><br>
<br>If anyone else has time to review the change as well, before I push it out, that would be great. I'm still new to this code base, and I don't want to break anything.<br><br>Ken<br><br><br><div class="gmail_quote">
On Mon, Jul 2, 2012 at 8:37 AM, ken cochrane <span dir="ltr"><<a href="mailto:kencochrane@gmail.com" target="_blank">kencochrane@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Jannis,</div><div><br></div>I started looking at this the other day, and I haven't had a chance to fix it because the Amazon datacenter outage took all of my time the past few days.<div><br></div><div>Here is what I found out.</div>
<div><br></div><div><a href="http://b.pypi.python.org/" style="line-height:18px;text-align:left;color:rgb(0,136,204);font-size:13px;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;text-decoration:none" target="_blank">b.pypi.python.org</a><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"> lives on GAE and it's currently stuck, and looking at the logs I figured out what went wrong, but I'm not sure how to fix it. [3] See log snippet at the end of the email.</span><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Basically there is a python package called '__past__' (see [0] link below) that is causing the sync process to break because we are trying to use the project name as the key_name for the Product model [1], and GAE model key_name's can't contain underscores [2]. </div>
<div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
I'm not sure how to fix the issue without possibly breaking other things. My first thought was to remove the underscores, but that might break something else, or conflict with another project with a similar name. I wrote to Martin who gave me the following advice.</div>
<div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="font-weight:bold;text-align:left;white-space:nowrap">From "Martin v. Löwis":</span></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="font-weight:bold;text-align:left;white-space:nowrap"><br></span></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Renaming/escaping sounds good. I'd check if there is any string that<br>can be used in a GAE key name, but not be used in a PyPI package name.<br>If not, standard escaping needs to be applied: a prefix of "dunder"<br>
is added to any package whose name starts and ends with __, as well<br>as to any package whose name starts with "dunder".<br></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
When looking at all child nodes, remove "dunder" from any name;<br>when doing lookups by name, escape as above.<br></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
If you do find a character/string that can be in a key name but<br>not in a package name, just escape the string with that name -<br>no need to worry about escaping the escape character. However it<br>may be that the only possible choice is "/" (which I know cannot<br>
appear in a package name).</blockquote><div><br></div><div>I looked through most of the pypi code and I think the only character you can't have is "/", all other characters look like they work. </div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">So, I know what is causing it, we just need to fix the issue, test it, and roll out the fix. I was planning on doing it this past weekend but thanks to AWS, I didn't have any time to work on it. If anyone has any free time, feel free to take over / help. Just let others know so there isn't any duplicate effort.</div>
<div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
Let me know if you have any questions.</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
Ken Cochrane</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
Footnotes:</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
[0] <a href="http://pypi.python.org/pypi/__past__/0.0.1.dev" style="color:rgb(17,85,204)" target="_blank">http://pypi.python.org/pypi/__past__/0.0.1.dev</a></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">[1] <a href="https://bitbucket.org/loewis/pypi-appengine/src/fa6596a427e1/fetch.py#cl-62" style="color:rgb(17,85,204)" target="_blank">https://bitbucket.org/loewis/pypi-appengine/src/fa6596a427e1/fetch.py#cl-62</a></div>
<div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">[2] Information about model key_names</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<a href="https://developers.google.com/appengine/docs/python/datastore/modelclass#Model" style="color:rgb(17,85,204)" target="_blank">https://developers.google.com/appengine/docs/python/datastore/modelclass#Model</a></div>
<div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><dt style="margin:0.33em 0px 0px 1.5em;padding:0px;border:0px;font-style:inherit;font-size:1.15em;font-family:inherit;vertical-align:baseline">
<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-size:0.95em;font-family:'Droid Sans Mono',monospace;vertical-align:baseline;letter-spacing:-0.5px">key_name</span></dt><dd style="margin:0px 0px 1.5em 2.5em;padding:0px;border:0px;font-style:inherit;font-family:inherit;vertical-align:baseline">
<p style="vertical-align:baseline;line-height:21px;color:rgb(51,51,51);font-family:Arial,sans-serif;margin:0px 0px 1.5em;border:0px;padding:0px">The key name for the entity. The name becomes part of the primary key. If <code style="margin:0px;padding:0px;border:0px;font-size:1em;font-family:'Droid Sans Mono',monospace;vertical-align:baseline;line-height:1.5;color:rgb(0,112,0)">None</code>, a system-generated numeric ID is used for the key.</p>
<p style="vertical-align:baseline;line-height:21px;color:rgb(51,51,51);font-family:Arial,sans-serif;margin:0px 0px 1.5em;border:0px;padding:0px">The value for <code style="margin:0px;padding:0px;border:0px;font-size:1em;font-family:'Droid Sans Mono',monospace;vertical-align:baseline;line-height:1.5;color:rgb(0,112,0)">key_name</code> must not be of the form <code style="margin:0px;padding:0px;border:0px;font-size:1em;font-family:'Droid Sans Mono',monospace;vertical-align:baseline;line-height:1.5;color:rgb(0,112,0)">__*__</code>.</p>
</dd></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">[1]</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
Log snippet.</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><ol style="margin:0px;padding:0px;border:0px;list-style:none;font-family:Arial,sans-serif;line-height:21px;text-align:left">
<li style="margin:0px 0px 0px 3px;padding:0px 0px 0px 17px;border:0px;line-height:1.5em;list-style:none"><h5 style="margin:0px;padding:0.4em 0px 0px;border:0px;font-size:1em;font-weight:normal;white-space:nowrap"><span style="margin:0px 0.3em">2012-06-28 06:45:18.222</span><span style="margin:0px 0.3em;color:rgb(102,102,102)"></span></h5>
<pre style="white-space:pre-wrap;word-wrap:break-word;padding:0.3em 0px 0.5em 1.5em;margin-top:0px;margin-bottom:0px;font-family:'Courier New'">step package '__past__'</pre></li><li style="margin:0px 0px 0px 3px;padding:0px 0px 0px 17px;border:0px;line-height:1.5em;list-style:none">
<h5 style="margin:0px;padding:0.4em 0px 0px;border:0px;font-size:1em;font-weight:normal;white-space:nowrap"><span title="Logging Severity: Error" style="display:block;float:left;min-height:1.2em;width:1.2em;line-height:1.2;text-align:center;text-transform:capitalize;font-weight:bold;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;background-color:rgb(255,153,0);margin:0px 0.3em">E</span><span style="margin:0px 0.3em">2012-06-28 06:45:18.778</span><span style="margin:0px 0.3em;color:rgb(102,102,102)"></span></h5>
<pre style="white-space:pre-wrap;word-wrap:break-word;padding:0.3em 0px 0.5em 1.5em;margin-top:0px;margin-bottom:0px;font-family:'Courier New'">illegal name in key path element: __past__
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/base/data/home/apps/pypi/3.358089379617981219/handlers.py", line 171, in get
self.response.out.write(fetch.cron())
File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 293, in cron
return step()
File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 259, in step
actions[action](m, todo, param)
File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 91, in package
data = simple_page(m, name)
File "/base/data/home/apps/pypi/3.358089379617981219/fetch.py", line 70, in simple_page
obj.put()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1074, in put
return datastore.Put(self._entity, **kwargs)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 579, in Put
return PutAsync(entities, **kwargs).get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1579, in __put_hook
self.check_rpc_success(rpc)
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1216, in check_rpc_success
raise _ToDatastoreError(err)
BadRequestError: illegal name in key path element: __past__</pre></li></ol></div><div><div class="h5">
<br><br><div class="gmail_quote">On Mon, Jul 2, 2012 at 8:11 AM, Jannis Leidel <span dir="ltr"><<a href="mailto:jannis@leidel.info" target="_blank">jannis@leidel.info</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 02.05.2011, at 22:10, Martin v. Löwis <<a href="mailto:martin@v.loewis.de" target="_blank">martin@v.loewis.de</a>> wrote:<br>
<br>
> Am 02.05.2011 19:24, schrieb Jannis Leidel:<br>
>><br>
>> On 02.05.2011, at 18:12, Maurits van Rees wrote:<br>
>><br>
>>> Hi,<br>
>>><br>
>>> I noticed that some distributions are not on all mirrors. For example<br>
>>> <a href="http://a.pypi.python.org/simple/plone.app.referenceablebehavior/" target="_blank">http://a.pypi.python.org/simple/plone.app.referenceablebehavior/</a><br>
>>> has 0.1 and 0.2 (last one released 30 April)<br>
>>> but 0.2 is missing from<br>
>>> <a href="http://b.pypi.python.org/simple/plone.app.referenceablebehavior/" target="_blank">http://b.pypi.python.org/simple/plone.app.referenceablebehavior/</a><br>
>>><br>
>>> Same for c and d. Ah, no: those two have it now. I know for sure that at least d did not have it five minutes ago. And this version has been released two days ago, so it should have been slightly faster. :-)<br>
>><br>
>> Hm, d doesn't seem to have the file on disk even thought it's on the simple page, see <a href="http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/" target="_blank">http://d.pypi.python.org/packages/source/p/plone.app.referenceablebehavior/</a><br>
>><br>
>> Martin: Anything I can do to make sure this doesn't happen again?<br>
><br>
> As the starting point, we should figure out why it happened in<br>
> the first place - it shouldn't have, of course. Most likely,<br>
> it's a bug :-)<br>
<br>
Looks like <a href="http://b.pypi.python.org" target="_blank">http://b.pypi.python.org</a> is out of date again: <a href="http://www.pypi-mirrors.org" target="_blank">http://www.pypi-mirrors.org</a><br>
<br>
Can we do something about that?<br>
<br>
Jannis<br>
<br>
_______________________________________________<br>
Catalog-SIG mailing list<br>
<a href="mailto:Catalog-SIG@python.org" target="_blank">Catalog-SIG@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/catalog-sig" target="_blank">http://mail.python.org/mailman/listinfo/catalog-sig</a><br>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>