<div dir="ltr"><div class="gmail_default" style="font-size:small">Hi Thomas,</div><div class="gmail_extra"><br><div class="gmail_quote">2017-05-20 13:23 GMT-04:00 Thomas Kluyver <span dir="ltr"><<a href="mailto:thomas@kluyver.me.uk" target="_blank">thomas@kluyver.me.uk</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div><div>Hi Luis,<br></div>
<div><br></div>
<div>Awesome, thanks for this :-). It was me posting before about indexing PyPI.<br></div>
<div><br></div>
<div>I'm intrigued: how do you keep it up to date using Travis? When I looked into this, I was pretty sure you need to download every package to index it. Do you have some way to only download the new releases? Or is Travis able to download every package every day? Or have you found another way round it?<br></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-size:small">I divided the index processing alphabetically, so that each letter is processed in a separate travis job. I also placed memory and time limits to avoid abusing Travis. The first run it has to download each package until it reaches the maximum time limit for each job, which is 40min. The next time, the script will only process packages that have been updated since the last run.</div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div></div>
<div><br></div>
<div>Does the index only include the latest version of each package, or does it also include older versions? The wifi on the train I'm on at the moment isn't fast enough to download 60 MB to find out. ;-)<br></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-size:small">It only includes the current versions.</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div></div>
<div><br></div>
<div>Does your indexing tool prefer to use wheels or sdists? Is it capable of using either for packages which don't have both available? Do you do anything to cope with modules which may be included for one platform but not another?<br></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-size:small">It supports ['.whl', '.egg', '.zip', '.tgz', '.tar.gz', '.tar.bz2'] formats, and it extracts the data using any available.<br><br>I wasn't aware of the fact that some modules may be on one platform and not in another. I guess there's room for improvement.</div></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div></div>
<div><br></div>
<div>I'm excited to see someone actually doing this!</div></div></blockquote><div><br><div class="gmail_default">Thank you. I made this because I wanted to have an app that guessed python dependencies from code by scaning module imports and then looking up the Index. That app is called Pip Sala Bim and you can check it out here:<br><br><a href="https://github.com/LuisAlejandro/pipsalabim">https://github.com/LuisAlejandro/pipsalabim</a><br></div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><span class="gmail-m_8511876957271830242gmail-HOEnZb"><font color="#888888"><br></font></span></div><span class="gmail-m_8511876957271830242gmail-HOEnZb"><font color="#888888">
<div><br></div>
<div>Thomas<br></div></font></span><div><div class="gmail-m_8511876957271830242gmail-h5">
<div><br></div>
<div><br></div>
<div>On Sat, May 20, 2017, at 03:01 AM, Luis Alejandro Martínez Faneyth wrote:<br></div>
</div></div><blockquote type="cite"><div><div class="gmail-m_8511876957271830242gmail-h5"><div dir="ltr"><div style="font-size:small">Hi everyone,<br></div>
<div style="font-size:small"><br></div>
<div style="font-size:small"><div>I'm new to this list but I've been reading some threads in the archive.<br></div>
<div><br></div>
<div>Around february, an idea about indexing modules from PyPI packages was brought up. I've been working on something similar for quite a while.<br></div>
<div><br></div>
<div>PyPIContents is an index of PyPI packages that lists its modules and command line scripts in JSON format, like this:<br></div>
<div><br></div>
<div><br></div>
<div><span class="gmail-m_8511876957271830242gmail-m_7611481223433706030colour" style="color:rgb(0,0,0)"><span class="gmail-m_8511876957271830242gmail-m_7611481223433706030font" style="font-family:monospace">[</span></span><br></div>
</div>
<div style="font-size:small"><div><span class="gmail-m_8511876957271830242gmail-m_7611481223433706030colour" style="color:rgb(0,0,0)"><span class="gmail-m_8511876957271830242gmail-m_7611481223433706030font" style="font-family:monospace">...</span></span><br></div>
<pre style="color:rgb(0,0,0);white-space:pre-wrap"> "1337": {
"cmdline": [],
"modules": [
"1337",
"1337.1337"
],
"version": "1.0.0"
},<br></pre><pre style="color:rgb(0,0,0);white-space:pre-wrap">...<br></pre><pre style="color:rgb(0,0,0);white-space:pre-wrap">]
<br></pre><pre style="color:rgb(0,0,0);white-space:pre-wrap"><br></pre></div>
<div><div>You can check it out here:<br></div>
<div><br></div>
<div><a href="https://github.com/LuisAlejandro/pypicontents" target="_blank">https://github.com/LuisAlejand<wbr>ro/pypicontents</a><br></div>
</div>
<div><br></div>
<div>And some use cases:<br></div>
<div><div><br></div>
<div><a href="https://github.com/LuisAlejandro/pypicontents#use-cases" target="_blank">https://github.com/LuisAlejand<wbr>ro/pypicontents#use-cases</a><br></div>
</div>
<div><br></div>
<div>The actual index lives here, its around 60MB:<br></div>
<div><br></div>
<div><div><a href="https://raw.githubusercontent.com/LuisAlejandro/pypicontents/contents/pypi.json" target="_blank">https://raw.githubusercontent.<wbr>com/LuisAlejandro/pypicontents<wbr>/contents/pypi.json</a><br></div>
<div><br></div>
<div>Is updated daily with the help of Travis:<br></div>
<div><br></div>
<div><a href="https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml" target="_blank">https://github.com/LuisAlejand<wbr>ro/pypicontents/blob/contents/<wbr>.travis.yml</a><br></div>
<div><br></div>
<div>Anyway, I hope is useful and I'll be around for any comments or questions.<br></div>
</div>
<div><br></div>
<div>Cheers!<br></div>
<div><br></div>
<div><div><div dir="ltr"><div><span></span><br></div>
<div><div dir="ltr"><div><span></span><br></div>
<div><span>Luis Alejandro Martínez Faneyth</span><br></div>
<div><span>Blog: <a href="http://huntingbears.com.ve" target="_blank">http://huntingbears.com.ve</a></span><br></div>
<div><span>Github: <a href="http://github.com/LuisAlejandro" target="_blank">http://github.com/LuisAlejandr<wbr>o</a></span><br></div>
<div><span>Twitter: <a href="http://twitter.com/LuisAlejandro" target="_blank">http://twitter.com/LuisAlejand<wbr>ro</a></span><br></div>
<div><span></span><br></div>
<div><span>CODE IS POETRY</span><br></div>
</div>
</div>
<div><span></span><br></div>
</div>
</div>
</div>
</div>
</div></div><span class="gmail-m_8511876957271830242gmail-"><div><u>______________________________<wbr>_________________</u><br></div>
<div>Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br></div>
<div><a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/distutils-sig</a><br></div>
</span></blockquote><div><br></div>
</div>
<br>______________________________<wbr>_________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/distutils-sig</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_8511876957271830242gmail_signature"><div dir="ltr"><span><div><div dir="ltr"><div>Luis Alejandro Martínez Faneyth</div><div>Blog: <a href="http://huntingbears.com.ve" target="_blank">http://huntingbears.com.ve</a></div><div>Github: <a href="http://github.com/LuisAlejandro" target="_blank">http://github.com/<wbr>LuisAlejandro</a></div><div>Twitter: <a href="http://twitter.com/LuisAlejandro" target="_blank">http://twitter.com/<wbr>LuisAlejandro</a></div><div><br></div><div>CODE IS POETRY</div></div></div></span></div></div>
</div></div>