FASTEN and dependency analysis at call graph level
Feb. 13, 2020
11:01 a.m.
Last week I heard for the first time about the research project FASTEN https://www.fasten-project.eu/. "FASTEN stands for Fine-Grained Analysis of Software Ecosystems as Networks."
instead of analyzing dependencies at the package level, we will analyze them at the call graph level! This will allow us to be super precise when we are tracking dependencies, when we do change impact analysis, when we recommend clients to update packages etc. It will also open the door to new sophisticated applications, e.g. licensing compliance, dependency risk profiling and data-driven API evolution.
That's from the blog post by Georgios Gousios, the PI, at http://www.gousios.gr/blog/Introducing-Fasten.html . More info: https://www.fasten-project.eu/view/Main/Introduction https://www.fasten-project.eu/view/Main/Overview https://www.fasten-project.eu/view/Main/Contacts And people who are interested in dynamically and statically analyzing call graphs in Python may be interested in "Graph Schema and its representation" in https://www.fasten-project.eu/view/Main/Deliverables . I've sent a note to FASTEN inviting the team to come talk about their project here on distutils-sig, because FASTEN's site says they aim to eventually integrate into PyPI -- I'm not 100% sure whether that means "create a service that people can use WITH PyPI" or "get FASTEN's work incorporated into pypi.org". -- Sumana Harihareswara pip project manager on contract with Python Software Foundation Changeset Consulting https://changeset.nyc
July 2020
1:37 p.m.
Hi Sumana, Thank you for reaching out to the project team. We do want to be integrated to package managers such as pip, and currently, we are in the early stages to develop the integration plan. It would be great to have feedback and collaboration with the python community on how our approach could be since the goal of the project is linked to several distinct ongoing discussions and PEPs. Please find more details below: Project overall concept: FASTEN stands for Fine-Grained Analysis of Software Ecosystems as Networks. We aim to make software ecosystems more robust by making package management more intelligent. The core idea that FASTEN relies upon is the creation of an ecosystem-wide Fine-Grained Call Graph (FGCG), at the function level. FASTEN will perform sophisticated analyses of i) security vulnerability propagation, ii) licensing compliance, and iii) dependency risk profiles. The result is a versioned, ecosystem-level call graph, that not only solves the issues identified above but also both opens the doors to advanced applications and challenges the current state of the art in graph storage and processing. To facilitate adoption, FASTEN will bring those analyses to the hands of developers by integrating the analysis service to popular package managers, for the Java, C, and Python programming languages. More info here: https://www.fasten-project.eu/view/Main/Overview pip integration proposal: A central aspect of the project is the promise to bring the power of the analyses afforded by the fine-grained ecosystem approach to the hands of the developers. Moreover, the package manager plug-ins will allow Continuous Integration servers and other components in Continuous Delivery pipelines to automate tasks such as compliance and security checks while a project is built (even behind corporate firewalls). For those reasons, we will create a plugin for pip and it will include the ability to produce call graphs compatible with FASTEN and instrument the host virtual machine to produce dynamic call graphs. The plug-in will interact with the FASTEN’s REST API. An example session with a FASTEN-enabled package manager can be seen below: ``` > pip list docutils (0.10) Jinja2 (2.7.2) MarkupSafe (0.18) > pip check-security Jinja2 (2.7.2) has known vulnerabilities (your project is affected!) Update to version >=2.7.3 (will not break your project) > pip test-upgrade Jinja2 --version 2.8 Upgrading to Jinja2 2.8 will break the following methods: myproject.foo() myproject.bar() > pip what-breaks --delete myproject.foo The following direct dependencies will break if you *delete* function foo() * projectA: 15 methods use foo() * projectB: 10 methods use foo() 632 indirect dependencies will fail to work. > pip test --upload-dyngraph ............15 Tests run OK! Dynamic call graph at: myproject.dot Uploading dynamic call graph to FASTEN ``` There is currently no such knowledge base in existence, let alone one with the level of detail that FASTEN aims for. Through integrations with existing services (e.g. GitHub) and developer tools (e.g. mvn and pip), the FASTEN Knowledge Base will make OSS software ecosystems better as a whole by increasing developer awareness and minimizing manual work that developers have to do to keep their projects up to date. The FASTEN consortium foresees that the Knowledge Base and its open REST API, will lead to a flourishing ecosystem of tools and services that take advantage of it; for this reason, FASTEN will work to create a community of contributing users around it. Our github: https://github.com/fasten-project/fasten FASTEN Overview presentation at OW2online20 conference: https://bittube.video/videos/watch/e16ff292-cfbe-4b44-86e2-0c1b040d5f9c -- Ingrid Sena Software Engineer @ Endocode AG Endocode AG, Brückenstraße 5A, 10179 Berlin +49 30 1206 4472 | info@endocode.com | www.endocode.com Vorstand: Lisa Nöth (Vorsitzende), Sebastian Sucker Aufsichtsratsvorsitzender: Mirko Boehm Registergericht: Amtsgericht Charlottenburg - HRB 150748 B
3:04 p.m.
Hi Ingrid, I think this could be an extremely valuable tool for anyone in need of maintaining a Python application. IMHO, being able to identify risks of upgrade errors based on type signatures of the public API of the dependencies we are using would be a game changer. Security checks would be also useful, but I believe there are already tools that can help with that. Regarding integration with pip: this would be nice, but I don't believe this is mandatory. The current generation of tools that help upgrade and pin dependencies (pip-tools, Poetry...) provide their own CLI, instead of extending pip. So both options are possible. Regards, S. On Thu, Jul 16, 2020 at 3:37 PM Ingrid Sena <job@ingridsena.com.br> wrote: > Hi Sumana, > Thank you for reaching out to the project team. > > We do want to be integrated to package managers such as pip, and > currently, we are in the early stages to develop the integration plan. It > would be great to have feedback and collaboration with the python community > on how our approach could be since the goal of the project is linked to > several distinct ongoing discussions and PEPs. > > Please find more details below: > > Project overall concept: > FASTEN stands for Fine-Grained Analysis of Software Ecosystems as Networks. > We aim to make software ecosystems more robust by making package > management more intelligent. The core idea that FASTEN relies upon is the > creation of an ecosystem-wide Fine-Grained Call Graph (FGCG), at the > function level. FASTEN will perform sophisticated analyses of i) security > vulnerability propagation, ii) licensing compliance, and iii) dependency > risk profiles. The result is a versioned, ecosystem-level call graph, that > not only solves the issues identified above but also both opens the doors > to advanced applications and challenges the current state of the art in > graph storage and processing. To facilitate adoption, FASTEN will bring > those analyses to the hands of developers by integrating the analysis > service to popular package managers, for the Java, C, and Python > programming languages. > > More info here: https://www.fasten-project.eu/view/Main/Overview > > pip integration proposal: > A central aspect of the project is the promise to bring the power of the > analyses afforded by the fine-grained ecosystem approach to the hands of > the developers. Moreover, the package manager plug-ins will allow > Continuous Integration servers and other components in Continuous Delivery > pipelines to automate tasks such as compliance and security checks while a > project is built (even behind corporate firewalls). For those reasons, we > will create a plugin for pip and it will include the ability to produce > call graphs compatible with FASTEN and instrument the host virtual machine > to produce dynamic call graphs. The plug-in will interact with the FASTEN’s > REST API. An example session with a FASTEN-enabled package manager can be > seen below: > > ``` > > pip list > docutils (0.10) > Jinja2 (2.7.2) > MarkupSafe (0.18) > > > pip check-security > Jinja2 (2.7.2) has known vulnerabilities (your project is affected!) > Update to version >=2.7.3 (will not break your project) > > > pip test-upgrade Jinja2 --version 2.8 > Upgrading to Jinja2 2.8 will break the following methods: > myproject.foo() > myproject.bar() > > > pip what-breaks --delete myproject.foo > The following direct dependencies will break if you *delete* function foo() > * projectA: 15 methods use foo() > * projectB: 10 methods use foo() > 632 indirect dependencies will fail to work. > > > pip test --upload-dyngraph > ............15 Tests run OK! > Dynamic call graph at: myproject.dot > Uploading dynamic call graph to FASTEN > ``` > > > There is currently no such knowledge base in existence, let alone one with > the level of detail that FASTEN aims for. Through integrations with > existing services (e.g. GitHub) and developer tools (e.g. mvn and pip), the > FASTEN Knowledge Base will make OSS software ecosystems better as a whole > by increasing developer awareness and minimizing manual work that > developers have to do to keep their projects up to date. The FASTEN > consortium foresees that the Knowledge Base and its open REST API, will > lead to a flourishing ecosystem of tools and services that take advantage > of it; for this reason, FASTEN will work to create a community of > contributing users around it. > > Our github: https://github.com/fasten-project/fasten > FASTEN Overview presentation at OW2online20 conference: > https://bittube.video/videos/watch/e16ff292-cfbe-4b44-86e2-0c1b040d5f9c > > -- > Ingrid Sena > Software Engineer @ Endocode AG > > Endocode AG, Brückenstraße 5A, 10179 Berlin > +49 30 1206 4472 | info@endocode.com | www.endocode.com > > Vorstand: Lisa Nöth (Vorsitzende), Sebastian Sucker > Aufsichtsratsvorsitzender: Mirko Boehm > > Registergericht: Amtsgericht Charlottenburg - HRB 150748 B > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/ZCV7OTM5WADBFE7XV6NRQC46PWCRHYYP/ > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
1711
Age (days ago)
1865
Last active (days ago)
2 comments
3 participants
participants (3)
-
Ingrid Sena -
Stéfane Fermigier -
Sumana Harihareswara