Hi Sylvain, First of all thanks for the quick answer. I tried to create a dummy project (the real one is confidential and not mine to give to the community) and it helped me narrow down my two issues. The ImportError: No module named project is linked to the relative import, and I could not reproduce the error in a dummy project. I finally found it, it has to do with my namespaces packages. What I mean by namespaces is in my python library I dispose of several packages like this : company.mongodb company.redis company.hbase And also : company.project.core company.project.app.dashboard company.project.app.security Where company and project are all the same things, this allows me and my team to easily identify each package and what it is supposed to do. We try to foster the open-source spirit in my company, hence the company top-level namespace. It is still internal but at least we share it across all teams :) Going back to the issue, if one package of the namespace is installed, then apparently pylint is looking directly inside the python path and can't find my package since it is not yet installed. I have the same issue with the "python setup.py develop" command, so I guess I am doing namespacing wrong, or it is not well defined in python ? The only thing I do is adding *_import__('pkg_resources').declare_namespace(__name__) *in the __init__.py of the namespace package. If I run pylint in another environment without any packages installed, it works like a charm (except I have a bunch of violations for missing imports). Do you want me to open an issue on bitbucket explaining all that or do you think I am doing something I am not supposed to do ? Cheers François 2014-05-16 14:02 GMT+02:00 Sylvain Thénault :

On 16 mai 11:16, François Vanderkelen wrote:

...

Hi experts !

Hi François,

...

I am struggling with Pylint with an issue that might be a problem coming from my code. Since I am relatively new to python (~1year) I would like to have your input on the matter.

I am trying to split a big python project I am doing for my company into several packages, allowing me to update each part without impacting the others. Doing that, we decided to go with Jenkins to improve our code quality which lead me to implement pylint for code violations checks.

Currently my build process is the following :

- Create a new virtualenv with Python version 2.7.6 - Install all dependencies needed by my package - Download the last version of the package from the master branch in our internal gitlab - Run nosetests with coverage reports - Run pylint and export the result to have some violations reports in Jenkins.

At first I had a lot of pylint errors since I wasn't installing all dependencies, resulting in something like 200 import errors ... Now it is reduced to some 60 errors but when I look a the build output log, I can see that pylint is actually crashing in the middle of the process.

************* Module company.project.core.security.app company/project/core/security/app.py:1: [C0111(missing-docstring), ] Missing module docstring load_entry_point('pylint==1.2.1', 'console_scripts', 'pylint')() File "/var/lib/jenkins/.pyenv/versions/2.7.6/lib/python2.7/site-packages/pylint/__init__.py", line 577, in _search_zip raise ImportError('No module named %s' % '.'.join(modpath)) ImportError: No module named project

Would you create an issue in pylint's bitbucket tracker for this? In any case it shouldn't crash like that?

...

This is how my package is constructed :

company/ -- __init__.py -- project/ -- __init__.py -- core/ -- __init__.py -- app.py -- security/ -- __init__.py -- app.py

In company.project.core.security.app I am trying to import company.project.core.app which contains some BaseClass for my project using:

from ..app import BaseApplication

This is probably another issue that deserves a ticket as well.

...

This is working fine with python, although pylint is crying out loud on this specific line. I tried to replace the relative import by core.app which worked fine with pylint but my code is not working anymore. I tried also to use the full path (company.project.core.app) but I have the exact same import error.

Be aware that both company and project are just namespace packages.

what do you call namespace package ? Do you mean part of them are installed in different directories?

...

Also I am forced to use the path to run pylint because it looks like my top-level folder is not a package (even if it contains a __init__ file).

pylint company.project.core No module named project.core (fatal)

and this work if you set a PYTHONPATH?

...

pylint company/project/core is working.

When I try to run pylint directly from the top-level package, I have the exact same result as from the path.

A workaround I can try is to install the package before running pylint on it but I reckon since my cwd is supposed to be added to the path I can't see how it would help (although I have other company packages installed).

It should work.

[syt@orion ~]$ mkdir x [syt@orion ~]$ mkdir x/y [syt@orion ~]$ touch x/__init__.py [syt@orion ~]$ touch x/y/__init__.py [syt@orion ~]$ pylint -rn x.y ************* Module x.y C: 1, 0: Missing module docstring (missing-docstring)

...

What do you think I should do ? Is my design a poorly chosen one and I don't have any solution ?

At a quick glance I would say you seem to abuse a bit from packages, leading to a deep structure which may be over-complicated. For instance, do you really want a 'company' first level package? Anyway, that doesn't seem to be the cause of your problems with pylint. At least, it shouldn't.

-- Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42) Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services CubicWeb, the semantic web framework: http://www.cubicweb.org

I tried multiple things, first even if I install the package before running pylint it won't work except on one of my machines. So I tried to narrow it down with setuptools : Working with 0.9.8 Not working with 2.2 (ImportError) Not working with 3.6 (ImportError) I guess setuptools changed something concerning namespaces packages, bug or not this is another story. I'll keep looking. Thanks a lot Sylvain 2014-05-16 15:01 GMT+02:00 Sylvain Thénault :

...

Hi Sylvain,

First of all thanks for the quick answer. I tried to create a dummy

...

(the real one is confidential and not mine to give to the community) and it helped me narrow down my two issues.

The ImportError: No module named project is linked to the relative import, and I could not reproduce the error in a dummy project. I finally found it, it has to do with my namespaces packages.

What I mean by namespaces is in my python library I dispose of several packages like this :

company.mongodb company.redis company.hbase

And also :

company.project.core company.project.app.dashboard company.project.app.security

Where company and project are all the same things, this allows me and my team to easily identify each package and what it is supposed to do. We

...

to foster the open-source spirit in my company, hence the company top-level namespace. It is still internal but at least we share it across all teams :)

Going back to the issue, if one package of the namespace is installed,

On 16 mai 14:42, François Vanderkelen wrote: project try then

...

apparently pylint is looking directly inside the python path and can't find my package since it is not yet installed. I have the same issue with the "python setup.py develop" command, so I guess I am doing namespacing wrong, or it is not well defined in python ?

The only thing I do is adding *_import__('pkg_resources').declare_namespace(__name__) *in the __init__.py of the namespace package.

If I run pylint in another environment without any packages installed, it works like a charm (except I have a bunch of violations for missing imports).

Do you want me to open an issue on bitbucket explaining all that or do you think I am doing something I am not supposed to do ?

I think what you're doing is right.

The crash has already been supported:

https://bitbucket.org/logilab/pylint/issue/203/importing-namespace-packages-...

It seems to me that the namespace package issue is related to recent version of setuptools and has also been reported but can't find it right away.

-- Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42) Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services CubicWeb, the semantic web framework: http://www.cubicweb.org

Hi François, by chance, would you be in EuroPython next week ? I intend to work on this namespace package problem there. Cheers On 01 juillet 11:22, François Vanderkelen wrote:

Hi Sylvain, experts.

I tried to work a little more on this and I found out why it "worked" with setuptools 0.9.8.

Turns out it did not, it is just when you start with a clean environement (not containing any of your namespaced packages) pylint is running properly and flagging every import of related namespace packages as missing.

Although the good part is that it is matching properly relatives import, you cannot have multiple dependencies to packages from the same namespace.

I am pretty sure that namespace usage is "erratic" in python since even Setuptools does not seem to respect PEP420, but I am not sure the fact the Pylint block on this comes from setuptools.

I think is is worth fixing but I am not sure where to start :)

2014-05-16 15:57 GMT+02:00 François Vanderkelen < vanderkelen.francois@gmail.com>:

...

I tried multiple things, first even if I install the package before running pylint it won't work except on one of my machines.

So I tried to narrow it down with setuptools :

Working with 0.9.8 Not working with 2.2 (ImportError) Not working with 3.6 (ImportError)

I guess setuptools changed something concerning namespaces packages, bug or not this is another story. I'll keep looking.

Thanks a lot Sylvain

2014-05-16 15:01 GMT+02:00 Sylvain Thénault :

On 16 mai 14:42, François Vanderkelen wrote:

...

...

Hi Sylvain,

First of all thanks for the quick answer. I tried to create a dummy project (the real one is confidential and not mine to give to the community) and it helped me narrow down my two issues.

The ImportError: No module named project is linked to the relative import, and I could not reproduce the error in a dummy project. I finally found it, it has to do with my namespaces packages.

What I mean by namespaces is in my python library I dispose of several packages like this :

company.mongodb company.redis company.hbase

And also :

company.project.core company.project.app.dashboard company.project.app.security

Where company and project are all the same things, this allows me and my team to easily identify each package and what it is supposed to do. We try to foster the open-source spirit in my company, hence the company top-level namespace. It is still internal but at least we share it across all teams :)

Going back to the issue, if one package of the namespace is installed, then apparently pylint is looking directly inside the python path and can't find my package since it is not yet installed. I have the same issue with the "python setup.py develop" command, so I guess I am doing namespacing wrong, or it is not well defined in python ?

The only thing I do is adding *_import__('pkg_resources').declare_namespace(__name__) *in the __init__.py of the namespace package.

If I run pylint in another environment without any packages installed, it works like a charm (except I have a bunch of violations for missing imports).

Do you want me to open an issue on bitbucket explaining all that or do you think I am doing something I am not supposed to do ?

I think what you're doing is right.

The crash has already been supported:

https://bitbucket.org/logilab/pylint/issue/203/importing-namespace-packages-...

It seems to me that the namespace package issue is related to recent version of setuptools and has also been reported but can't find it right away.

-- Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42) Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services CubicWeb, the semantic web framework: http://www.cubicweb.org

-- Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42) Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services CubicWeb, the semantic web framework: http://www.cubicweb.org