There seems to be something wrong with your install of either Sphinx or numpydoc. If it's missing completely, the traceback says:

  Could not import extension numpydoc (exception: No module named 'numpydoc')

rather than complaining about "Directive". Possibly a mixup with versions? Also note that you're saying that you see numpydoc in the sphinx directory - did you mean in site-packages instead? That's where it should be:

You're right, I meant in the site-packages directory. For me it's here: '/home/aditya/scipy-dev/lib/python3.5/site-packages/numpydoc/__init__.py'.
/home/aditya/scipy-dev is my virtualenv for scipy. It uses python3.5.

I tried making the docs outside the virtual environment, I'm getting the same error. For reference, I'm using:
  • python 3.5.2
  • Sphinx version 1.7.3
  • numpydoc version 0.8.0
  • scipy version 1.0.0 / version 1.2.0.dev0+3988d44 (virtualenv)