Mailman 3 Search results for "top-pypi-packages"

[Python-Dev] Re: What to do about invalid escape sequences

by Matt Billenstein

On Tue, Aug 06, 2019 at 04:32:04PM +0000, Matt Billenstein wrote: > Perhaps those packages could be flagged now via pylint and problems raised with > the respective package maintainers before the actual 3.8 release? Checking the > top 100 or top 1000 packages on PyPI? fwiw, ran pylint on the top 100 pypi pkgs from: https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json The list of packages is pretty small: https://gist.github.com/mattbillenstein/ad862d032b8575f8d6e08384850f2223 but some have quite a few errors... m -- Matt Billenstein matt(a)vazor.com http://www.vazor.com/

5 years, 3 months

[Python-Dev] Tool to search in the source code of PyPI top 5000 projects

by Victor Stinner

Hi, I wrote two scripts based on the work of INADA-san's work to (1) download the source code of the PyPI top 5000 projects (2) search for a regex in these projects (compressed source archives). You can use these tools if you work on an incompatible Python or C API change to estimate how many projects are impacted. The HPy project created a Git repository for a similar need (latest update in June 2021): https://github.com/hpyproject/top4000-pypi-packages There are also online services for code search: * GitHub: https://github.com/search * https://grep.app/ (I didn't try it yet) * Debian: https://codesearch.debian.net/ (1) Dowload Script: https://github.com/vstinner/misc/blob/main/cpython/download_pypi_top.py Usage: download_pypi_top.py PATH It uses this JSON file: https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.min.js… From this service: https://hugovk.github.io/top-pypi-packages/ At December 1, on 5000 projects, it only downloads 4760 tarball and ZIP archives: I guess that 240 projects don't provide a source archive. It takes around 5,2 GB of disk space. (2) Code search First, I used the fast and nice "ripgrep" tool with the command "rg -zl REGEX path/*.{zip,gz,bz2,tgz}" (-z searchs in ZIP and tarball archives). But it doesn't show the path inside the archive and it searchs in files generated by Cython whereas I wanted to ignore these files. So I wrote a short Python script which decompress tarball and ZIP archive in memory and looks for a regex: https://github.com/vstinner/misc/blob/main/cpython/search_pypi_top.py Usage: search_pypi_top.py "REGEX" output_filename The code to parse command line option is hardcoded and pypi_dir = "PYPI-2021-12-01-TOP-5000" are hardcoded :-D It ignores files generated by Cython and .so binary files (Linux dynamic libraries). While "rg" is very fast, my script is very slow. But I don't care, once the regex is written, I only need to search for the regex once, I can wait 10-15 min ;-) I prefer to wait longer and have a more accurate result. Also, there is room for enhancement, like running multiple jobs in different processes or threads. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 years, 11 months

[capi-sig] Re: Announcing breaking changes (was: Re: Py_TYPE() and Py_SIZE() become static inline functions)

by Petr Viktorin

On 09. 09. 21 23:36, Antonio Cuni wrote: > On Thu, Sep 9, 2021 at 5:06 PM Marc-Andre Lemburg <mal(a)egenix.com> wrote: > >> >> Analyze the top PyPI packages, identify the ones which would break >> and send the maintainers an email explaining the change. >> > > for this particular issue, I have a repository which contains the C code in > the top 4000 PyPI packages: > https://github.com/hpyproject/top4000-pypi-packages > > the repo is imperfect: to keep the size small it contains only the C code > (thus you can't build the packages from there), I manually removed some > duplicated/vendored/autogenerated files and it's just a snapshot of a > certain point in time. However, it is very useful in combination with grep > to have some idea of how often a certain API and/or pattern is used in the > real world. Sadly, "top 4000 PyPI packages" is very biased towards widely-used projects and the public projects -- ones least likely to have issues, because the community (of widely-used projects) or the author of the CPython change (for public projects) can submit fixes early. It would be sad if CPython focused only on successful, public projects; but I fear that's where we'll end up if we use this heuristic when evaluating changes.

3 years, 2 months

[Python-Dev] Tips: Searching deprecated API usage.

by Inada Naoki

Hi, folks. After 3.9 becomes beta, I am searching deprecated APIs we can remove in Python 3.10. I want to share how I am checking how the API is not used. Please teach me if you know an easy and better approach to find deprecated API usage. ## Sourcegraph Github code search is not powerful enough and there is a lot of noise. (e.g. many people copy CPython source code). On the other hand, Sourcegraph only searches from major repositories, and has powerful filtering. This is an example of `PyEval_ReleaseLock` search. https://sourcegraph.com/search?q=PyEval_ReleaseLock+file:.*%5C.%28cc%7Ccxx%… ## Top 4000 packages You can download a list of top 4000 PyPI packages in JSON format from this site. https://hugovk.github.io/top-pypi-packages/ I used this script to download sdist packages from the JSON file. https://github.com/methane/notes/blob/master/2020/wchar-cache/download_sdis… Note that this script doesn't download packages without sdist (e.g. only universal wheel). It is because I have searched Python/C API. We can reduce the pain of the removal by fixing most of top 4000 packages. Regards, -- Inada Naoki <songofacandy(a)gmail.com>

4 years, 4 months

[Python-Dev] Re: Tool to search in the source code of PyPI top 5000 projects

by Steve Dower

FTR, I don't consider the top projects on PyPI to be representative of our user base, and *especially* not representative of people compiling native modules. This is not a good way to evaluate the impact of breaking changes. It would be far safer to assume that every change is going to break someone and evaluate: * how will they find out that upgrading Python will cause them to break * how will they find out where that break occurs * how will they find out how to fix it * how will they manage that fix across multiple releases * how will we explain that upgrading and fixing breaks is better for *them* than staying on the older version This last one is particularly important, as many large organisations (anecdotally) seem to have settled on Python 3.7 for a while now. Inevitably, this means they're all going to be faced with a painful time when it comes to an upgrade, and every little change we add on is going to hurt more. Every extra thing that needs fixing is motivation to just rewrite in a new language with more hype (and the promise of better compatibility... which I won't comment specifically on, but I suspect they won't manage it any better than us ;) ). This is not the case for the top PyPI projects. They incrementally update and crowdsource fixes, often from us. The pain is distributed to the level of permanent background noise, which sucks in its own way, but is ultimately not representative of much of our user base. So by all means, use this tool for checking stuff. But it's not a substitute for justifying every incompatible change in its own right. /rant Cheers, Steve On 12/2/2021 11:44 PM, Victor Stinner wrote: > Hi, > > I wrote two scripts based on the work of INADA-san's work to (1) > download the source code of the PyPI top 5000 projects (2) search for > a regex in these projects (compressed source archives). > > You can use these tools if you work on an incompatible Python or C API > change to estimate how many projects are impacted. > > The HPy project created a Git repository for a similar need (latest > update in June 2021): > https://github.com/hpyproject/top4000-pypi-packages > > There are also online services for code search: > > * GitHub: https://github.com/search > * https://grep.app/ (I didn't try it yet) > * Debian: https://codesearch.debian.net/ > > > (1) Dowload > > Script: > https://github.com/vstinner/misc/blob/main/cpython/download_pypi_top.py > > Usage: download_pypi_top.py PATH > > It uses this JSON file: > https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.min.js… > > From this service: > https://hugovk.github.io/top-pypi-packages/ > > At December 1, on 5000 projects, it only downloads 4760 tarball and > ZIP archives: I guess that 240 projects don't provide a source > archive. It takes around 5,2 GB of disk space. > > > (2) Code search > > First, I used the fast and nice "ripgrep" tool with the command "rg > -zl REGEX path/*.{zip,gz,bz2,tgz}" (-z searchs in ZIP and tarball > archives). But it doesn't show the path inside the archive and it > searchs in files generated by Cython whereas I wanted to ignore these > files. > > So I wrote a short Python script which decompress tarball and ZIP > archive in memory and looks for a regex: > https://github.com/vstinner/misc/blob/main/cpython/search_pypi_top.py > > Usage: search_pypi_top.py "REGEX" output_filename > > The code to parse command line option is hardcoded and pypi_dir = > "PYPI-2021-12-01-TOP-5000" are hardcoded :-D > > It ignores files generated by Cython and .so binary files (Linux > dynamic libraries). > > While "rg" is very fast, my script is very slow. But I don't care, > once the regex is written, I only need to search for the regex once, I > can wait 10-15 min ;-) I prefer to wait longer and have a more > accurate result. Also, there is room for enhancement, like running > multiple jobs in different processes or threads. > > Victor >

2 years, 11 months

[capi-sig] Re: Announcing breaking changes (was: Re: Py_TYPE() and Py_SIZE() become static inline functions)

by Antonio Cuni

On Thu, Sep 9, 2021 at 5:06 PM Marc-Andre Lemburg <mal(a)egenix.com> wrote: > > Analyze the top PyPI packages, identify the ones which would break > and send the maintainers an email explaining the change. > for this particular issue, I have a repository which contains the C code in the top 4000 PyPI packages: https://github.com/hpyproject/top4000-pypi-packages the repo is imperfect: to keep the size small it contains only the C code (thus you can't build the packages from there), I manually removed some duplicated/vendored/autogenerated files and it's just a snapshot of a certain point in time. However, it is very useful in combination with grep to have some idea of how often a certain API and/or pattern is used in the real world.

3 years, 2 months

[Python-Dev] Re: Tips: Searching deprecated API usage.

by Victor Stinner

Thanks for your advice! I confirm that raw GitHub code search is painful with all copies of CPython code base :-( I copied your notes in my https://pythondev.readthedocs.io/test_next_python.html list :-) victor Le dim. 5 juil. 2020 à 07:55, Inada Naoki <songofacandy(a)gmail.com> a écrit : > > Hi, folks. > > After 3.9 becomes beta, I am searching deprecated APIs we can remove > in Python 3.10. > I want to share how I am checking how the API is not used. > > Please teach me if you know an easy and better approach to find > deprecated API usage. > > ## Sourcegraph > > Github code search is not powerful enough and there is a lot of noise. > (e.g. many people copy CPython source code). > On the other hand, Sourcegraph only searches from major repositories, > and has powerful filtering. > This is an example of `PyEval_ReleaseLock` search. > > https://sourcegraph.com/search?q=PyEval_ReleaseLock+file:.*%5C.%28cc%7Ccxx%… > > > ## Top 4000 packages > > You can download a list of top 4000 PyPI packages in JSON format from this site. > https://hugovk.github.io/top-pypi-packages/ > > I used this script to download sdist packages from the JSON file. > https://github.com/methane/notes/blob/master/2020/wchar-cache/download_sdis… > > Note that this script doesn't download packages without sdist (e.g. > only universal wheel). > It is because I have searched Python/C API. > We can reduce the pain of the removal by fixing most of top 4000 packages. > > > Regards, > -- > Inada Naoki <songofacandy(a)gmail.com> > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/USDJAUA… > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death.

4 years, 4 months

[capi-sig] Re: Announcing breaking changes (was: Re: Py_TYPE() and Py_SIZE() become static inline functions)

by Victor Stinner

On Thu, Sep 9, 2021 at 5:05 PM Marc-Andre Lemburg <mal(a)egenix.com> wrote: > Since this ML is not really on the radar of many extension writers, > how about creating a C API change announcement list where breaking > changes are posted or, even better: > > Analyze the top PyPI packages, identify the ones which would break > and send the maintainers an email explaining the change. Sure, I did that whenever I could. First, my conclusion: it's kind of sad that there is no "reliable" and "automated" way to rebuild *all* PyPI projects to count how many projects would be broken by an incompatible change, when working on a change. I do this work manually. Because this task is manual, I'm more in favor of doing a quick estimate of the risk first, then push the change, and consider reverting if too many problems are discovered during the development phase. I know that it's not perfect. I also know that some people consider that the C API is perfect and that it must not be changed :-) But there are *reasons* to "fix" it :-) -- I copied Inada-san tool to download PyPI top (5000) projects: https://github.com/vstinner/misc/blob/main/cpython/download_pypi_top.py You can then use tools like "rg" to grep regex patterns directly in tarballs and ZIP archives. See Inada-san notes about that: https://github.com/methane/notes/tree/master/2020/wchar-cache I usually use it as an estimation on how many projects will be impacted outside the PyPI most popular projects. -- Moreover, at Red Hat, we are rebuilding the whole Fedora operating system (technically, we try to only rebuild Python packages) with newer Python during alpha and then beta releases. We are usually the first ones to detect Python packages impacted by incompatible changes. That's how we decided to revert a few incompatible changes in Python 3.9 for example: https://lwn.net/Articles/811369/ That's also how we decided to revert the Py_TYPE() change in Python 3.10. -- There is also the code search on GitHub, but it's not easy to skip the big number of copies of the CPython source code in results. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

3 years, 2 months

[capi-sig]Re: Split the big C API problem into multiple PEPs

by Nick Coghlan

On Sat, 23 Feb 2019 at 10:08, Steve Dower <steve.dower(a)python.org> wrote: > (And specifically on PEP A - I hear a lot of people say "check the top > 10/20/30 C extensions", but I don't actually know what they are? Even > just a list of them would be great! And I bet 10 of them are the ones > included in CPython ;) ) While PyPI download stats have their issues, they're mostly sufficient for this purpose: https://hugovk.github.io/top-pypi-packages/ A dozen or so entries from the top 100 that include binary extensions: - simplejson - numpy - lxml - pyyaml - cryptography - protobuf - pandas - pyopenssl - scipy - psycopg2 - grpcio - wrapt - matplotlib - pynacl And some entries worth mentioning that don't show up in frequent download lists (suggesting something about the nature of automated CI environments for rich client GUI applications): - sip - pyqt5 - pyside - pygobject - wxpython - pyobjc-core - pywin32 Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 years, 8 months

[Python-Dev] Re: Having Sorted Containers in stdlib?

by Marc-Andre Lemburg

On 12.11.2021 17:10, Steven D'Aprano wrote: > On Fri, Nov 12, 2021 at 10:07:13AM -0500, Paul Ganssle wrote: > >> I knew about sortedcontainers and I also don't remember ever seeing a >> situation where I needed one or recommended its use. > > We have a very odd situation where apparently sortedcontainers is one > of the most well-known, popular, most heavily downloaded libraries on > PyPI. According to here: > > https://hugovk.github.io/top-pypi-packages/ > > sortedcontainers is the 290th most popular package on PyPI, ahead of > such luminaries as pylint, black, selenium, mypy, django and nose. > > And yet, nobody(?) admits to either using it or knowing what it could be > used for. How very curious :-/ Those download stats can be misleading. Packages are often pulled in as a dependency of other packages which are popular or used a lot in CI/CD setups. Perhaps there's a reverse dependency graph we could use to find out why the package is downloaded this often. I remember having seen a project which does this, but have lost the URL. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 12 2021) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

3 years

Search results for query "top-pypi-packages"