To install sphinxcontrib.katex into your Python virtual environment run:

$ pip install sphinxcontrib-katex

If you want to pre-render the math by running Javascript on your server instead of running it in the browsers of the users, you have to install katex on your server and add it to your path:

$ npm install katex
$ PATH="${PATH}:$(pwd)/node_modules/.bin"