[pypy-dev] TypeError: expected string, got NoneType object with setuptools and co with pypy 4

[Nagy, Attila]

Hi,

After installing pypy 4.0.1 on FreeBSD (from ports), I get the above 
exception when trying to install setuptools, or when installing anything 
with pip (after pypy -m ensurepip).

This all worked with the previous version (2.6).

I could find this problem elsewhere:
http://stackoverflow.com/questions/34566676/failed-to-install-pip-for-pypy-on-ubuntu
so it doesn't seem to be related to FreeBSD.

An example full trace:
# pypy -m pip install --upgrade pip
You are using pip version 6.1.1, however version 8.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip
   Exception:
   Traceback (most recent call last):
     File "/usr/local/pypy-4.0/site-packages/pip/basecommand.py", line 
246, in main
       status = self.run(options, args)
     File "/usr/local/pypy-4.0/site-packages/pip/commands/install.py", 
line 342, in run
       requirement_set.prepare_files(finder)
     File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 
345, in prepare_files
       functools.partial(self._prepare_file, finder))
     File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 
290, in _walk_req_to_install
       more_reqs = handler(req_to_install)
     File "/usr/local/pypy-4.0/lib_pypy/_functools.py", line 42, in __call__
       return self._func(*(self._args + fargs), **fkeywords)
     File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 
487, in _prepare_file
       download_dir, do_download, session=self.session,
     File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 827, 
in unpack_url
       session,
     File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 673, 
in unpack_http_url
       from_path, content_type = _download_http_url(link, session, temp_dir)
     File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 887, 
in _download_http_url
       with open(file_path, 'wb') as content_file:
   TypeError: expected string, got NoneType object


And the one for setuptools:
# pypy setup.py install
running install
Checking .pth file support in /usr/local/pypy-4.0/site-packages/
/usr/local/bin/pypy -E -c pass
TEST PASSED: /usr/local/pypy-4.0/site-packages/ appears to support .pth 
files
running bdist_egg
running egg_info
writing setuptools.egg-info/PKG-INFO
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
writing requirements to setuptools.egg-info/requires.txt
writing top-level names to setuptools.egg-info/top_level.txt
Traceback (most recent call last):
   File "setup.py", line 169, in <module>
     dist = setuptools.setup(**setup_params)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/core.py", line 
151, in setup
     dist.run_commands()
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
953, in run_commands
     self.run_command(cmd)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
972, in run_command
     cmd_obj.run()
   File "/tmp/setuptools-19.4/setuptools/command/install.py", line 67, 
in run
     self.do_egg_install()
   File "/tmp/setuptools-19.4/setuptools/command/install.py", line 109, 
in do_egg_install
     self.run_command('bdist_egg')
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/cmd.py", line 326, 
in run_command
     self.distribution.run_command(command)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
972, in run_command
     cmd_obj.run()
   File "/tmp/setuptools-19.4/setuptools/command/bdist_egg.py", line 
152, in run
     self.run_command("egg_info")
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/cmd.py", line 326, 
in run_command
     self.distribution.run_command(command)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
972, in run_command
     cmd_obj.run()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 186, 
in run
     self.find_sources()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 209, 
in find_sources
     mm.run()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 293, 
in run
     self.add_defaults()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 322, 
in add_defaults
     sdist.add_defaults(self)
   File "/tmp/setuptools-19.4/setuptools/command/sdist.py", line 100, in 
add_defaults
     self.filelist.append(fn)
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 236, 
in append
     if self._safe_path(path):
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 256, 
in _safe_path
     u_path = unicode_utils.filesys_decode(path)
   File "/tmp/setuptools-19.4/setuptools/unicode_utils.py", line 31, in 
filesys_decode
     return path.decode(enc)
TypeError: expected string, got NoneType object

In this case the context is:
def filesys_decode(path):
     """
     Ensure that the given path is decoded,
     NONE when no expected encoding works
     """

     fs_enc = sys.getfilesystemencoding()
     if isinstance(path, six.text_type):
         return path

     for enc in (fs_enc, "utf-8"):
         try:
             return path.decode(enc)
         except UnicodeDecodeError:
             continue

On python 2.7, fs_enc here is:
Python 2.7.11 (default, Dec 20 2015, 01:15:21)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 
208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> repr(sys.getfilesystemencoding())
"'US-ASCII'"

While on pypy:
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Jan 16 2016, 
01:16:36)
[PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> repr(sys.getfilesystemencoding())
'None'

Returning None here is fine according to the docs, but the above code 
snippet doesn't handle the TypeError, which it gets when doing 
path.decode(None).
How this could work?

On pypy 2.6 on the same machine:
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jun 12 2015, 
19:25:58)
[PyPy 2.6.0] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> 'test'.decode(None)
u'test'
No exception!

On pypy 4.0.1:
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Dec 10 2015, 
01:17:03)
[PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> 'test'.decode(None)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: expected string, got NoneType object

Unhandled TypeError exception!

python 2.7 works the same way:
>>> 'test'.decode(None)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: decode() argument 1 must be string, not None

Which installs setuptools just fine (although it doesn't get to the 
above snippet, because it doesn't return None to 
sys.getfilesystemencoding()).
If I set fs_enc to None in the above snippet, even python 2.7 fails.

So the key here (if I'm not completely lost) seems to be not returning 
None to the getfilesystemencoding to fix broken(?) software.

What do you think about this?

Thanks,