<div dir="ltr">On Sun, Jun 23, 2013 at 6:11 PM, Donald Stufft <span dir="ltr"><<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><div><div class="h5"><div>On Jun 23, 2013, at 9:36 AM, Alex Clark <<a href="mailto:aclark@aclark.net" target="_blank">aclark@aclark.net</a>> wrote:</div>
<br><blockquote type="cite">Hi Noah,<br><br><br>Noah Kantrowitz <noah <at> <a href="http://coderanger.net" target="_blank">coderanger.net</a>> writes:<br><br><blockquote type="cite"><br><br>On Jun 22, 2013, at 10:33 PM, anatoly techtonik wrote:<br>
<br><blockquote type="cite">On Fri, Jun 14, 2013 at 5:05 PM, anatoly techtonik <techtonik <at><br></blockquote></blockquote><a href="http://gmail.com" target="_blank">gmail.com</a>> wrote:<br><blockquote type="cite">
<blockquote type="cite">Could you, please, share the log format+example, so that we can<br></blockquote></blockquote>experiment with it?<br><blockquote type="cite"><blockquote type="cite"><br>ping<br></blockquote><br>Additional assistance is not required for this project. Thank you for your<br>
</blockquote>interest, and stay tuned for<br><blockquote type="cite">future updates.<br></blockquote><br><br>WAT.<br><br><br>Can you explain why "additional assistance is not required for this<br>project"? AFAICT:<br>
<br>- Shit is broken (i.e. download counts)<br>- At first, there was no plans to fix it (which I accepted as a cost to<br>enable the CDN)<br>- Now there are plans to fix it, but no additional assistance is needed? If<br>
that were the case I'd expect download counts to be fixed. Or at the very<br>
least, I'd expect a "no thanks we don't need help, but here is where we are<br>at [ explanation of situation ]". Not "we don't need help, we'll be in<br>touch". I think the latter is a disservice to the community, which (I<br>
assume) you especially don't want to be responsible for given the<br>tremendously positive service you've done by enabling the CDN in the first<br>place.)<br><br><br>We get it: download counts were not as important as CDN. But they *are*<br>
important, so let's keep talking about how to fix them.<br></blockquote><div><br></div></div></div><div><a href="https://en.wikipedia.org/wiki/Brooks%27_law" target="_blank">https://en.wikipedia.org/wiki/Brooks%27_law</a></div>
<div><br></div><div>To further expand. Right now the biggest issue preventing the CDN download counts is a less than ideal configuration on the VM hosting PyPI. I've been working on (and almost have completed) a Chef cookbook that will allow us to easily deploy PyPI to a second VM (and gracefully do the switch over) with minimal downtime.</div>
<div><br></div><div>However there is an issue with launching VM's at the moment that OSUOL is looking into which means even with a completed cookbook we couldn't launch a VM at the moment anyways. If we wanted to correct the configuration in place it would require downtime.</div>
</div></div></blockquote><div><br></div></div>I understand the pressure you're working, but I believe Brooks law doesn't apply here, because you're working on server configuration problem, while I propose to help with actual download counting problem. It is, of course, interesting to know the details of what are you trying to achieve to see where people can plug in. As I understand the problem you're trying to solve right now is (1) how to store the log or to receive it. If you're solving some problem (0) and don't have time for (1) yet - tell us the input data - so that somebody could prepare some variants. Problem (2), is to pipe the log to analyzer. I guess it will transform from "something" you got in (1) into stream of lines. Problem (3) is count processing - consume line stream, process, organize in batches and write to PyPI store. Working on (3) doesn't require any of the (0)-(2) steps to be complete.<br>
<br>I am pushing this because I doubt that Chef configuration is an easy and fast process. At least a year ago my experience with Chef in a limited time constraint was negative (which should not come as a surprise, because I mostly see only things that are negative). I started with zero after Noah's talk at PyCon, but within 2 months I failed to provide a solution for automatic server deployment for just a couple of servers with a little bit different configuration for web and DB server than official cookbooks support. This post at <a href="http://lumberjaph.net/devops/2012/11/27/ansible-and-chef.html">http://lumberjaph.net/devops/2012/11/27/ansible-and-chef.html</a> summarizes my experience. I am not sure you will be allowed to use Ansible for this stuff, but I'd give it a try at least for a prototype configuration.<br>
<br><br></div></div>