Hi Sylvain,

First of all thanks for the quick answer. I tried to create a dummy project (the real one is confidential and not mine to give to the community) and it helped me narrow down my two issues.

The ImportError: No module named project is linked to the relative import, and I could not reproduce the error in a dummy project. I finally found it, it has to do with my namespaces packages.

What I mean by namespaces is in my python library I dispose of several packages like this :

company.mongodb
company.redis
company.hbase

And also :

company.project.core
company.project.app.dashboard
company.project.app.security

Where company and project are all the same things, this allows me and my team to easily identify each package and what it is supposed to do. We try to foster the open-source spirit in my company, hence the company top-level namespace. It is still internal but at least we share it across all teams :)

Going back to the issue, if one package of the namespace is installed, then apparently pylint is looking directly inside the python path and can't find my package since it is not yet installed. I have the same issue with the "python setup.py develop" command, so I guess I am doing namespacing wrong, or it is not well defined in python ? 

The only thing I do is adding _import__('pkg_resources').declare_namespace(__name__) in the __init__.py of the namespace package.

If I run pylint in another environment without any packages installed, it works like a charm (except I have a bunch of violations for missing imports).

Do you want me to open an issue on bitbucket explaining all that or do you think I am doing something I am not supposed to do ?

Cheers

François


2014-05-16 14:02 GMT+02:00 Sylvain Thénault <sylvain.thenault@logilab.fr>:
On 16 mai 11:16, François Vanderkelen wrote:
> Hi experts !

Hi François,

> I am struggling with Pylint with an issue that might be a problem coming
> from my code. Since I am relatively new to python (~1year) I would like to
> have your input on the matter.
>
> I am trying to split a big python project I am doing for my company into
> several packages, allowing me to update each part without impacting the
> others. Doing that, we decided to go with Jenkins to improve our code
> quality which lead me to implement pylint for code violations checks.
>
> Currently my build process is the following :
>
> - Create a new virtualenv with Python version 2.7.6
> - Install all dependencies needed by my package
> - Download the last version of the package from the master branch in our
> internal gitlab
> - Run nosetests with coverage reports
> - Run pylint and export the result to have some violations reports in
> Jenkins.
>
> At first I had a lot of pylint errors since I wasn't installing all
> dependencies, resulting in something like 200 import errors ... Now it is
> reduced to some 60 errors but when I look a the build output log, I can see
> that pylint is actually crashing in the middle of the process.
>
> ************* Module company.project.core.security.app
> company/project/core/security/app.py:1: [C0111(missing-docstring), ]
> Missing module docstring
> load_entry_point('pylint==1.2.1', 'console_scripts', 'pylint')()
>   File "/var/lib/jenkins/.pyenv/versions/2.7.6/lib/python2.7/site-packages/pylint/__init__.py",
> line 577, in _search_zip
>     raise ImportError('No module named %s' % '.'.join(modpath))
> ImportError: No module named project

Would you create an issue in pylint's bitbucket tracker for this? In any case
it shouldn't crash like that?

> This is how my package is constructed :
>
> company/
> -- __init__.py
> -- project/
>    -- __init__.py
>    -- core/
>       -- __init__.py
>       -- app.py
>       -- security/
>          -- __init__.py
>          -- app.py
>
> In company.project.core.security.app I am trying to import
> company.project.core.app which contains some BaseClass for my project using:
>
> from ..app import BaseApplication

This is probably another issue that deserves a ticket as well.

> This is working fine with python, although pylint is crying out loud on
> this specific line. I tried to replace the relative import by core.app
> which worked fine with pylint but my code is not working anymore. I tried
> also to use the full path (company.project.core.app) but I have the exact
> same import error.
>
> Be aware that both company and project are just namespace packages.

what do you call namespace package ? Do you mean part of them are installed in
different directories?

> Also I am forced to use the path to run pylint because it looks like my
> top-level folder is not a package (even if it contains a __init__ file).
>
> pylint company.project.core
> No module named project.core (fatal)

and this work if you set a PYTHONPATH?

> pylint company/project/core is working.
>
> When I try to run pylint directly from the top-level package, I have the
> exact same result as from the path.
>
> A workaround I can try is to install the package before running pylint on
> it but I reckon since my cwd is supposed to be added to the path I can't
> see how it would help (although I have other company packages installed).

It should work.

[syt@orion ~]$ mkdir x
[syt@orion ~]$ mkdir x/y
[syt@orion ~]$ touch x/__init__.py
[syt@orion ~]$ touch x/y/__init__.py
[syt@orion ~]$ pylint -rn x.y
************* Module x.y
C:  1, 0: Missing module docstring (missing-docstring)

> What do you think I should do ? Is my design a poorly chosen one and I
> don't have any solution ?

At a quick glance I would say you seem to abuse a bit from packages, leading to
a deep structure which may be over-complicated. For instance, do you really want
a 'company' first level package? Anyway, that doesn't seem to be the cause of
your problems with pylint. At least, it shouldn't.

--
Sylvain Thénault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42)
Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
Développement logiciel sur mesure:       http://www.logilab.fr/services
CubicWeb, the semantic web framework:    http://www.cubicweb.org