<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On May 27, 2013, at 4:34 PM, Noah Kantrowitz <<a href="mailto:noah@coderanger.net">noah@coderanger.net</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><br>On May 27, 2013, at 1:20 PM, holger krekel wrote:<br><br><blockquote type="cite">On Mon, May 27, 2013 at 12:58 -0700, Noah Kantrowitz wrote:<br><blockquote type="cite">On May 27, 2013, at 12:18 PM, holger krekel wrote:<br><br><blockquote type="cite">On Mon, May 27, 2013 at 14:59 -0400, Donald Stufft wrote:<br><blockquote type="cite">On May 27, 2013, at 2:54 PM, holger krekel <<a href="mailto:holger@merlinux.eu">holger@merlinux.eu</a>> wrote:<br><br><blockquote type="cite">On Mon, May 27, 2013 at 13:50 -0400, Donald Stufft wrote:<br><blockquote type="cite">On May 27, 2013, at 12:39 PM, Donald Stufft <<a href="mailto:donald@stufft.io">donald@stufft.io</a>> wrote:<br><br><blockquote type="cite"><br>On May 27, 2013, at 8:08 AM, holger krekel <<a href="mailto:holger@merlinux.eu">holger@merlinux.eu</a>> wrote:<br><br><blockquote type="cite">Hi Noah, Donald, (CC also Richard, Christian),<br><br>i just checked with a test package and think we might have a cache<br>consistency / changelog API problem. It took me a while but here is <br>the basic thing: I uploaded a test package, changelog API reports it has<br>changed, then i go to its simple page, and some of the time the new release<br>file shows up, sometimes not.<br><br>Tools like bandersnatch, pep381 and devpi-server (and probably others)<br>use PyPI's changelog API to determine if there are changes. It seems<br>those changes are signalled faster than they become consistently accessible <br>through the CDN. This can lead to inconsistent mirrors because when <br>the CDN has the files there is no change event anymore. Such mirrors <br>are run by companies in-house so i think it's a real problem.<br><br>Even without mirroring there can be problems because installs are not<br>directly repeatable: "pip install XYZ>=2.0" can give you first 2.0.1,<br>then 2.0.0 a minute later. I had hoped that a particular ip address<br>sees things consistently.<br><br>I am not familiar with Fastly's caching properties -- can they notify<br>about the fact that a page/file is consistently up-to-date everywhere? <br>Or can the cache be globally invalidated for a particular page/file?<br>Any other ideas?<br><br>Failing customizing Fastly usage and also maybe for the short term,<br>is/could there be a special location provided by <a href="http://pypi.python.org">pypi.python.org</a> which<br>the above tools could use to get at the actual non-cached data? We<br>could then maybe mitigate the problem through updates of the respective tools.<br>That would at least solve the problem for one of my customers i think.<br><br>best,<br>holger<br><br><br>On Sun, May 26, 2013 at 10:34 -0700, Noah Kantrowitz wrote:<br><blockquote type="cite"></farnsworth><br><br>but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me.<br><br>--Noah<br><br></blockquote><br><br><br><blockquote type="cite">_______________________________________________<br>Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br><a href="http://mail.python.org/mailman/listinfo/distutils-sig">http://mail.python.org/mailman/listinfo/distutils-sig</a><br></blockquote><br>_______________________________________________<br>Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br><a href="http://mail.python.org/mailman/listinfo/distutils-sig">http://mail.python.org/mailman/listinfo/distutils-sig</a><br></blockquote><br>I mentioned it on twitter but might as well mention it here as well.<br><br>Currently there is no invalidation going on. The effect on the mirroring was unanticipated and I'm currently getting the invalidation API setup within PyPI.<br><br>-----------------<br>Donald Stufft<br>PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br><br>_______________________________________________<br>Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br><a href="http://mail.python.org/mailman/listinfo/distutils-sig">http://mail.python.org/mailman/listinfo/distutils-sig</a><br></blockquote><br><br><br>/simple/ Pages should now be immediately invalidated when a new package is released.<br></blockquote><br>thanks Donald. Looking at the implementation, i wonder what happens if <br>after ``self._conn.commit()`` a changelog API call arrives, returns changes<br>and a client uses it to retrieve changes before the fastly-purging takes <br>place. It's still a potential race-condition or am i missing something?<br><br>best,<br>holger<br><br><blockquote type="cite">-----------------<br>Donald Stufft<br>PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br><br></blockquote><br><br></blockquote><br><br>There's no way around a race condition.<br><br>``self._conn.commit()`` is what makes the changes available. If we purge prior to committing it then if someone hits the page between the purge and the self._conn.commit() then the client will see a page cached prior to the update (while the change log will appear to be updated). Essentially the same problem we have now.<br><br>The current implementation does mean that if a client happens to hit between the commit and the purge they'll see old data however that's pretty unlikely.<br></blockquote><br>Purging can take a second and also depends on the network connectivity <br>between <a href="http://pypi.python.org">pypi.python.org</a> and fastly's api to begin with. I am afraid <br>the race-condition is bound to happen and then hard to detect. <br><br>Not sure how exactly <a href="http://pypi.python.org">pypi.python.org</a> is deployed but could commit() use<br>a semaphore which also the changelog-APIs use so that the latter only<br>returns after purging (and them some) has happened? I don't think<br>mirrors would mind sometimes waiting a few seconds before the changelog* call<br>returns as long as the state is then consistent.<br><br>Lastly, i think introducing a bit of internal syncing overhead to commit()/<br>changelog should be ok because we have only few writes and hardly read load.<br></blockquote><br>Mirroring should not be affected by caching at all, as new packages mean new URLs (/pypi/name/version), so when you retrieve them there will be no cache issues. <br></blockquote><br>The simple/PROJ pages are changed, not newly created. (and yes,<br>new release files are not so much the problem because they are new<br>and thus retrieved from fastly on first access).<br></blockquote><br>Yes, pep381client is fundamentally incompatible with the future of PyPI's infrastructure. Sorry, this will not be changed at this point. If people would like to continue to operate mirrors, they will need to transition to use the API to access package information, fetch updated files, and rebuild any relevant index data. For example, this is how Donald's crate.io mirror operates. Using the current strategy of scraping the simple/ pages will continue to work, you just need to retry failed requests until they succeed (and check that the per-project pages match the version you expect from the change log, consider it a failure if they do not). This is just a stopgap though, and should not be considered a long-term solution.<br><br><blockquote type="cite"><br><blockquote type="cite">What I think you mean is this makes a race condition for pep381client,<br>however this is a bug in pep381client, not PyPI. If you would like to<br>submit a patch for a Paxos-based replication protocol, I'm sure Donald<br>and I would be happy to review it.<br></blockquote><br>I am a bit lost of what you are talking about here.<br><br>The move to CDN broke things that worked before. The changelog API<br>reported changes that could not be seen afterwards. This remains true<br>after Donald's changes which just make it less likely but not impossible<br>to happen.<br></blockquote><br>They worked effectively by accident, not because it was correct. Had I understood how backwards the pep381 systems are, I would have alerted you all sooner, I apologize for this lapse. I am happy to talk about how to correctly use the PyPI API with anyone that has questions, or discuss more advanced replication options that will be free of race conditions in a distributed (yes, PyPI is a distributed database now) environment.<br><br>--Noah<br><br></blockquote></div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">Just to assure folks. I do consider Mirroring a first class citizen and an important feature.</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><br></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">Better support will be coming for it. However this relatively minor issue requiring that clients get a little smarter to use the current system is a regrettable requirement to make installation from PyPI not suck for a large swathe of people.</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><br></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">-----------------</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">Donald Stufft</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div>
</div>
<br></body></html>